2021-02-11T09:34:47.078 INFO:root:teuthology version: 1.1.0-14fad317 2021-02-11T09:34:47.079 DEBUG:teuthology.run:Teuthology command: teuthology --name aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi --owner scheduled_aemerson@teuthology --description rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} --archive /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874503 --verbose -- /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874503/orig.config.yaml 2021-02-11T09:34:47.210 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-02-11T09:34:47.267 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874503 branch: wip-blurpletastic description: rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: aemerson@redhat.com first_in_suite: false job_id: '5874503' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: wip-blurpletastic ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync obj etag verify: true setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 5e5bb33c614118d912ca83c2e5f2be67b3acc04f wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 5e5bb33c614118d912ca83c2e5f2be67b3acc04f rgw: compression type: random frontend: civetweb rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: wip-blurpletastic sha1: 5e5bb33c614118d912ca83c2e5f2be67b3acc04f owner: scheduled_aemerson@teuthology priority: 70 repo: git://git.ceph.com/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 5e5bb33c614118d912ca83c2e5f2be67b3acc04f sleep_before_teardown: 0 suite: rgw suite_branch: wip-blurpletastic suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 5e5bb33c614118d912ca83c2e5f2be67b3acc04f targets: smithi026.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBDwsvOghUsw0AfpZm6OwXwfU7McUxVq6l7RtzNuUCIs2rKPswTlou++DeBtTVmRqJTIvOyW9OoQsjokpItJ041v5vF6BHqilMGhMlgYOzTf38D5IDODjNCE2gWbGEQWgT9RLoBsSxSboqj4JGkGb5RlJorJkG79sce4RT+vjRaDxX3SPXlEGhut2cND0DigGU2o95eMaY+el+7ro58L3cGymLBEQBgdqAG0ylRGOWaJs2IWRQefwZOOt9UMbQNWtvXOd8CQb6B/JVaqj5OAJoRkrbQY2E5V9/oS7kc8EH8EsNWHyStaHB9lSblIVsDAapREJ+ceLCtJZ4vSlLimBp smithi133.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDApmU2RgrDF6OBBSRN7KgTBoRZ18J5CDi+nwxXLLj06kaIh8EVEmOUck98z8UUzSnuJOxY3IzU7xPZwgg8CXnqTn3rdQl6qIh3PU0VReuWmSqZ07BCWiclTLDXQ0ldeDP/WUTKGHVR8FBsCEjOa1Ns7DP0txtTPVkjpRQhTW96V0oWdP9V5NcKrFpOKnkxrQANTim+ih1cMw5r9mBTQQT7IhHS1c5hwAe7JhKIVP+Ur6HhsCgltW9hfC4Wt686bNcI1HHEDA2SI7xQShPKbLq32LP59ifbRaSe5j8voANtgFnK/R2cSW/sonM55p7BRgZlNow4qATJS09nrT3XbdyF tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master timestamp: 2021-02-11_09:24:13 tube: smithi user: aemerson verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.29888 2021-02-11T09:34:47.268 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa; will attempt to use it 2021-02-11T09:34:47.273 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa/tasks 2021-02-11T09:34:47.273 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-02-11T09:34:47.275 INFO:teuthology.task.internal:Checking packages... 2021-02-11T09:34:47.311 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '5e5bb33c614118d912ca83c2e5f2be67b3acc04f' 2021-02-11T09:34:47.312 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-02-11T09:34:47.312 INFO:teuthology.packaging:ref: None 2021-02-11T09:34:47.312 INFO:teuthology.packaging:tag: None 2021-02-11T09:34:47.312 INFO:teuthology.packaging:branch: wip-blurpletastic 2021-02-11T09:34:47.312 INFO:teuthology.packaging:sha1: 5e5bb33c614118d912ca83c2e5f2be67b3acc04f 2021-02-11T09:34:47.312 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-blurpletastic 2021-02-11T09:34:47.483 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-735-g5e5bb33c-1bionic 2021-02-11T09:34:47.483 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-02-11T09:34:47.490 INFO:teuthology.task.internal:no buildpackages task found 2021-02-11T09:34:47.490 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-02-11T09:34:47.510 INFO:teuthology.task.internal:Saving configuration 2021-02-11T09:34:47.523 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-02-11T09:34:47.533 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-11T09:34:47.551 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-11 09:29:29.761533', 'description': '/home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874503', 'locked_by': 'scheduled_aemerson@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFB5ZWPfvTne9dCvv+Q5zI4eejd7dOE593K2VYqtT4BC', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi026.front.sepia.ceph.com'} 2021-02-11T09:34:47.568 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-11 09:29:29.761514', 'description': '/home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874503', 'locked_by': 'scheduled_aemerson@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJkm6k+39JaIXptOBp0Q7arSdf1UjBIk7l5msW3PbM7/', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi133.front.sepia.ceph.com'} 2021-02-11T09:34:47.568 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-02-11T09:34:47.576 INFO:teuthology.task.internal:roles: ubuntu@smithi026.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2021-02-11T09:34:47.576 INFO:teuthology.task.internal:roles: ubuntu@smithi133.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2021-02-11T09:34:47.576 INFO:teuthology.run_tasks:Running task console_log... 2021-02-11T09:34:47.656 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0a688152f0>, signals=[15]) 2021-02-11T09:34:47.656 INFO:teuthology.run_tasks:Running task internal.connect... 2021-02-11T09:34:47.667 INFO:teuthology.task.internal:Opening connections... 2021-02-11T09:34:47.667 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi026.front.sepia.ceph.com 2021-02-11T09:34:47.669 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T09:34:47.744 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi133.front.sepia.ceph.com 2021-02-11T09:34:47.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T09:34:47.820 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-02-11T09:34:47.828 DEBUG:teuthology.orchestra.run.smithi026:> uname -m 2021-02-11T09:34:47.949 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2021-02-11T09:34:47.949 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2021-02-11T09:34:47.998 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="Ubuntu" 2021-02-11T09:34:47.998 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-11T09:34:47.998 INFO:teuthology.orchestra.run.smithi026.stdout:ID=ubuntu 2021-02-11T09:34:47.998 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE=debian 2021-02-11T09:34:47.998 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-11T09:34:47.998 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="18.04" 2021-02-11T09:34:47.998 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-11T09:34:47.999 INFO:teuthology.orchestra.run.smithi026.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-11T09:34:47.999 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-11T09:34:47.999 INFO:teuthology.orchestra.run.smithi026.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-11T09:34:47.999 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_CODENAME=bionic 2021-02-11T09:34:47.999 INFO:teuthology.orchestra.run.smithi026.stdout:UBUNTU_CODENAME=bionic 2021-02-11T09:34:48.000 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2021-02-11T09:34:48.028 DEBUG:teuthology.orchestra.run.smithi133:> uname -m 2021-02-11T09:34:48.033 INFO:teuthology.orchestra.run.smithi133.stdout:x86_64 2021-02-11T09:34:48.034 DEBUG:teuthology.orchestra.run.smithi133:> cat /etc/os-release 2021-02-11T09:34:48.080 INFO:teuthology.orchestra.run.smithi133.stdout:NAME="Ubuntu" 2021-02-11T09:34:48.080 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-11T09:34:48.080 INFO:teuthology.orchestra.run.smithi133.stdout:ID=ubuntu 2021-02-11T09:34:48.080 INFO:teuthology.orchestra.run.smithi133.stdout:ID_LIKE=debian 2021-02-11T09:34:48.080 INFO:teuthology.orchestra.run.smithi133.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-11T09:34:48.080 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_ID="18.04" 2021-02-11T09:34:48.080 INFO:teuthology.orchestra.run.smithi133.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-11T09:34:48.081 INFO:teuthology.orchestra.run.smithi133.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-11T09:34:48.081 INFO:teuthology.orchestra.run.smithi133.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-11T09:34:48.081 INFO:teuthology.orchestra.run.smithi133.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-11T09:34:48.081 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_CODENAME=bionic 2021-02-11T09:34:48.081 INFO:teuthology.orchestra.run.smithi133.stdout:UBUNTU_CODENAME=bionic 2021-02-11T09:34:48.081 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server 2021-02-11T09:34:48.095 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-02-11T09:34:48.106 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-02-11T09:34:48.267 INFO:teuthology.task.internal:Checking for old test directory... 2021-02-11T09:34:48.267 DEBUG:teuthology.orchestra.run.smithi026:> test '!' -e /home/ubuntu/cephtest 2021-02-11T09:34:48.269 DEBUG:teuthology.orchestra.run.smithi133:> test '!' -e /home/ubuntu/cephtest 2021-02-11T09:34:48.274 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-02-11T09:34:48.285 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-02-11T09:34:48.286 DEBUG:teuthology.orchestra.run.smithi026:> test -z $(ls -A /var/lib/ceph) 2021-02-11T09:34:48.314 DEBUG:teuthology.orchestra.run.smithi133:> test -z $(ls -A /var/lib/ceph) 2021-02-11T09:34:48.343 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-02-11T09:34:48.386 INFO:teuthology.run_tasks:Running task kernel... 2021-02-11T09:34:48.404 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2021-02-11T09:34:48.404 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-02-11T09:34:48.404 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2021-02-11T09:34:48.410 INFO:teuthology.orchestra.run.smithi026.stdout:4.15.0-134-generic 2021-02-11T09:34:48.410 INFO:teuthology.task.kernel:Running kernel on smithi026: 4.15.0-134-generic 2021-02-11T09:34:48.411 DEBUG:teuthology.orchestra.run.smithi026:> sudo apt-get clean 2021-02-11T09:34:48.531 DEBUG:teuthology.orchestra.run.smithi026:> sudo apt-get update 2021-02-11T09:34:48.789 INFO:teuthology.orchestra.run.smithi026.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-02-11T09:34:48.811 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T09:34:48.870 INFO:teuthology.orchestra.run.smithi026.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-02-11T09:34:49.142 INFO:teuthology.orchestra.run.smithi026.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-02-11T09:34:49.223 INFO:teuthology.orchestra.run.smithi026.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,543 kB] 2021-02-11T09:34:49.351 INFO:teuthology.orchestra.run.smithi026.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,883 kB] 2021-02-11T09:34:49.668 INFO:teuthology.orchestra.run.smithi026.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [915 kB] 2021-02-11T09:34:49.687 INFO:teuthology.orchestra.run.smithi026.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [298 kB] 2021-02-11T09:34:49.692 INFO:teuthology.orchestra.run.smithi026.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [226 kB] 2021-02-11T09:34:49.696 INFO:teuthology.orchestra.run.smithi026.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [29.9 kB] 2021-02-11T09:34:49.697 INFO:teuthology.orchestra.run.smithi026.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,109 kB] 2021-02-11T09:34:49.767 INFO:teuthology.orchestra.run.smithi026.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [975 kB] 2021-02-11T09:34:49.773 INFO:teuthology.orchestra.run.smithi026.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,218 kB] 2021-02-11T09:34:49.790 INFO:teuthology.orchestra.run.smithi026.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [248 kB] 2021-02-11T09:34:49.809 INFO:teuthology.orchestra.run.smithi026.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [390 kB] 2021-02-11T09:34:49.820 INFO:teuthology.orchestra.run.smithi026.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [247 kB] 2021-02-11T09:34:49.827 INFO:teuthology.orchestra.run.smithi026.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [33.4 kB] 2021-02-11T09:34:49.828 INFO:teuthology.orchestra.run.smithi026.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,560 kB] 2021-02-11T09:34:49.897 INFO:teuthology.orchestra.run.smithi026.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,718 kB] 2021-02-11T09:34:49.998 INFO:teuthology.orchestra.run.smithi026.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [363 kB] 2021-02-11T09:34:50.006 INFO:teuthology.orchestra.run.smithi026.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [31.8 kB] 2021-02-11T09:34:50.006 INFO:teuthology.orchestra.run.smithi026.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [14.3 kB] 2021-02-11T09:34:50.007 INFO:teuthology.orchestra.run.smithi026.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [7,300 B] 2021-02-11T09:34:51.440 INFO:teuthology.orchestra.run.smithi026.stdout:Fetched 13.1 MB in 3s (4,893 kB/s) 2021-02-11T09:34:52.942 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2021-02-11T09:34:52.992 DEBUG:teuthology.orchestra.run.smithi026:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-11T09:34:53.072 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2021-02-11T09:34:53.341 INFO:teuthology.orchestra.run.smithi026.stdout:Building dependency tree... 2021-02-11T09:34:53.342 INFO:teuthology.orchestra.run.smithi026.stdout:Reading state information... 2021-02-11T09:34:53.429 INFO:teuthology.orchestra.run.smithi026.stderr:E: Unable to locate package linux-image-current-generic 2021-02-11T09:34:53.430 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-11T09:34:53.431 DEBUG:teuthology.orchestra.run.smithi026:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-11T09:34:53.509 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2021-02-11T09:34:53.768 INFO:teuthology.orchestra.run.smithi026.stdout:Building dependency tree... 2021-02-11T09:34:53.768 INFO:teuthology.orchestra.run.smithi026.stdout:Reading state information... 2021-02-11T09:34:53.956 INFO:teuthology.orchestra.run.smithi026.stdout:The following additional packages will be installed: 2021-02-11T09:34:53.956 INFO:teuthology.orchestra.run.smithi026.stdout: linux-generic linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-11T09:34:53.956 INFO:teuthology.orchestra.run.smithi026.stdout: linux-headers-generic linux-image-4.15.0-135-generic 2021-02-11T09:34:53.957 INFO:teuthology.orchestra.run.smithi026.stdout: linux-modules-4.15.0-135-generic linux-modules-extra-4.15.0-135-generic 2021-02-11T09:34:53.958 INFO:teuthology.orchestra.run.smithi026.stdout:Suggested packages: 2021-02-11T09:34:53.959 INFO:teuthology.orchestra.run.smithi026.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2021-02-11T09:34:53.959 INFO:teuthology.orchestra.run.smithi026.stdout:Recommended packages: 2021-02-11T09:34:53.959 INFO:teuthology.orchestra.run.smithi026.stdout: thermald 2021-02-11T09:34:53.994 INFO:teuthology.orchestra.run.smithi026.stdout:The following NEW packages will be installed: 2021-02-11T09:34:53.994 INFO:teuthology.orchestra.run.smithi026.stdout: linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-11T09:34:53.994 INFO:teuthology.orchestra.run.smithi026.stdout: linux-image-4.15.0-135-generic linux-modules-4.15.0-135-generic 2021-02-11T09:34:53.995 INFO:teuthology.orchestra.run.smithi026.stdout: linux-modules-extra-4.15.0-135-generic 2021-02-11T09:34:53.996 INFO:teuthology.orchestra.run.smithi026.stdout:The following packages will be upgraded: 2021-02-11T09:34:53.997 INFO:teuthology.orchestra.run.smithi026.stdout: linux-generic linux-headers-generic linux-image-generic 2021-02-11T09:34:54.176 INFO:teuthology.orchestra.run.smithi026.stdout:3 upgraded, 5 newly installed, 0 to remove and 23 not upgraded. 2021-02-11T09:34:54.176 INFO:teuthology.orchestra.run.smithi026.stdout:Need to get 66.0 MB of archives. 2021-02-11T09:34:54.176 INFO:teuthology.orchestra.run.smithi026.stdout:After this operation, 334 MB of additional disk space will be used. 2021-02-11T09:34:54.176 INFO:teuthology.orchestra.run.smithi026.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-135-generic amd64 4.15.0-135.139 [13.1 MB] 2021-02-11T09:34:55.179 INFO:teuthology.orchestra.run.smithi026.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-135-generic amd64 4.15.0-135.139 [8,019 kB] 2021-02-11T09:34:55.421 INFO:teuthology.orchestra.run.smithi026.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-135-generic amd64 4.15.0-135.139 [32.8 MB] 2021-02-11T09:34:56.379 INFO:teuthology.orchestra.run.smithi026.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.135.122 [1,868 B] 2021-02-11T09:34:56.379 INFO:teuthology.orchestra.run.smithi026.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.135.122 [2,592 B] 2021-02-11T09:34:56.379 INFO:teuthology.orchestra.run.smithi026.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135 all 4.15.0-135.139 [10.9 MB] 2021-02-11T09:34:56.654 INFO:teuthology.orchestra.run.smithi026.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135-generic amd64 4.15.0-135.139 [1,097 kB] 2021-02-11T09:34:56.707 INFO:teuthology.orchestra.run.smithi026.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.135.122 [2,488 B] 2021-02-11T09:34:57.197 INFO:teuthology.orchestra.run.smithi026.stdout:Fetched 66.0 MB in 3s (24.5 MB/s) 2021-02-11T09:34:57.328 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package linux-modules-4.15.0-135-generic. 2021-02-11T09:34:57.371 INFO:teuthology.orchestra.run.smithi026.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102079 files and directories currently installed.) 2021-02-11T09:34:57.374 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../0-linux-modules-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T09:34:57.425 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:34:58.959 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package linux-image-4.15.0-135-generic. 2021-02-11T09:34:58.971 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../1-linux-image-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T09:34:59.054 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:34:59.545 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-135-generic. 2021-02-11T09:34:59.558 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T09:34:59.595 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:35:03.545 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../3-linux-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T09:35:03.650 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking linux-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T09:35:03.983 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T09:35:04.103 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking linux-image-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T09:35:04.456 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package linux-headers-4.15.0-135. 2021-02-11T09:35:04.468 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../5-linux-headers-4.15.0-135_4.15.0-135.139_all.deb ... 2021-02-11T09:35:04.506 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-11T09:35:09.281 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package linux-headers-4.15.0-135-generic. 2021-02-11T09:35:09.303 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../6-linux-headers-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T09:35:09.347 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:35:11.831 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T09:35:11.951 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking linux-headers-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T09:35:12.288 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:35:12.848 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-11T09:35:12.957 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:35:13.392 INFO:teuthology.orchestra.run.smithi026.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-135-generic 2021-02-11T09:35:13.392 INFO:teuthology.orchestra.run.smithi026.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-135-generic 2021-02-11T09:35:13.526 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:35:14.028 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:35:14.154 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up linux-image-generic (4.15.0.135.122) ... 2021-02-11T09:35:14.279 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up linux-headers-generic (4.15.0.135.122) ... 2021-02-11T09:35:14.399 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up linux-generic (4.15.0.135.122) ... 2021-02-11T09:35:14.517 INFO:teuthology.orchestra.run.smithi026.stdout:Processing triggers for linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:35:14.695 INFO:teuthology.orchestra.run.smithi026.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-02-11T09:35:14.695 INFO:teuthology.orchestra.run.smithi026.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-135-generic 2021-02-11T09:35:29.578 INFO:teuthology.orchestra.run.smithi026.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-02-11T09:35:29.579 INFO:teuthology.orchestra.run.smithi026.stdout:Sourcing file `/etc/default/grub' 2021-02-11T09:35:29.604 INFO:teuthology.orchestra.run.smithi026.stdout:Generating grub configuration file ... 2021-02-11T09:35:30.061 INFO:teuthology.orchestra.run.smithi026.stdout:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-11T09:35:30.083 INFO:teuthology.orchestra.run.smithi026.stdout:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-11T09:35:30.479 INFO:teuthology.orchestra.run.smithi026.stdout:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-11T09:35:30.505 INFO:teuthology.orchestra.run.smithi026.stdout:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-11T09:35:31.948 INFO:teuthology.orchestra.run.smithi026.stdout:done 2021-02-11T09:35:32.362 DEBUG:teuthology.orchestra.run.smithi026:> dpkg -s linux-image-generic 2021-02-11T09:35:32.385 INFO:teuthology.orchestra.run.smithi026.stdout:Package: linux-image-generic 2021-02-11T09:35:32.385 INFO:teuthology.orchestra.run.smithi026.stdout:Status: install ok installed 2021-02-11T09:35:32.385 INFO:teuthology.orchestra.run.smithi026.stdout:Priority: optional 2021-02-11T09:35:32.385 INFO:teuthology.orchestra.run.smithi026.stdout:Section: kernel 2021-02-11T09:35:32.385 INFO:teuthology.orchestra.run.smithi026.stdout:Installed-Size: 17 2021-02-11T09:35:32.385 INFO:teuthology.orchestra.run.smithi026.stdout:Maintainer: Ubuntu Kernel Team 2021-02-11T09:35:32.385 INFO:teuthology.orchestra.run.smithi026.stdout:Architecture: amd64 2021-02-11T09:35:32.385 INFO:teuthology.orchestra.run.smithi026.stdout:Source: linux-meta 2021-02-11T09:35:32.385 INFO:teuthology.orchestra.run.smithi026.stdout:Version: 4.15.0.135.122 2021-02-11T09:35:32.385 INFO:teuthology.orchestra.run.smithi026.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-11T09:35:32.385 INFO:teuthology.orchestra.run.smithi026.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-11T09:35:32.385 INFO:teuthology.orchestra.run.smithi026.stdout:Recommends: thermald 2021-02-11T09:35:32.386 INFO:teuthology.orchestra.run.smithi026.stdout:Description: Generic Linux kernel image 2021-02-11T09:35:32.386 INFO:teuthology.orchestra.run.smithi026.stdout: This package will always depend on the latest generic kernel image 2021-02-11T09:35:32.386 INFO:teuthology.orchestra.run.smithi026.stdout: available. 2021-02-11T09:35:32.386 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-134-generic Expected: 4.15.0-135-generic 2021-02-11T09:35:32.386 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2021-02-11T09:35:32.392 INFO:teuthology.orchestra.run.smithi133.stdout:4.15.0-134-generic 2021-02-11T09:35:32.393 INFO:teuthology.task.kernel:Running kernel on smithi133: 4.15.0-134-generic 2021-02-11T09:35:32.393 DEBUG:teuthology.orchestra.run.smithi133:> sudo apt-get clean 2021-02-11T09:35:32.498 DEBUG:teuthology.orchestra.run.smithi133:> sudo apt-get update 2021-02-11T09:35:32.626 INFO:teuthology.orchestra.run.smithi133.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-02-11T09:35:32.740 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T09:35:32.818 INFO:teuthology.orchestra.run.smithi133.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-02-11T09:35:32.882 INFO:teuthology.orchestra.run.smithi133.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [915 kB] 2021-02-11T09:35:33.001 INFO:teuthology.orchestra.run.smithi133.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,543 kB] 2021-02-11T09:35:33.028 INFO:teuthology.orchestra.run.smithi133.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [298 kB] 2021-02-11T09:35:33.033 INFO:teuthology.orchestra.run.smithi133.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [226 kB] 2021-02-11T09:35:33.037 INFO:teuthology.orchestra.run.smithi133.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [29.9 kB] 2021-02-11T09:35:33.038 INFO:teuthology.orchestra.run.smithi133.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [975 kB] 2021-02-11T09:35:33.055 INFO:teuthology.orchestra.run.smithi133.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,109 kB] 2021-02-11T09:35:33.075 INFO:teuthology.orchestra.run.smithi133.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [248 kB] 2021-02-11T09:35:33.089 INFO:teuthology.orchestra.run.smithi133.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-02-11T09:35:33.217 INFO:teuthology.orchestra.run.smithi133.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,218 kB] 2021-02-11T09:35:33.501 INFO:teuthology.orchestra.run.smithi133.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,883 kB] 2021-02-11T09:35:33.602 INFO:teuthology.orchestra.run.smithi133.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [390 kB] 2021-02-11T09:35:33.624 INFO:teuthology.orchestra.run.smithi133.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [247 kB] 2021-02-11T09:35:33.630 INFO:teuthology.orchestra.run.smithi133.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [33.4 kB] 2021-02-11T09:35:33.631 INFO:teuthology.orchestra.run.smithi133.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,718 kB] 2021-02-11T09:35:33.667 INFO:teuthology.orchestra.run.smithi133.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,560 kB] 2021-02-11T09:35:33.724 INFO:teuthology.orchestra.run.smithi133.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [363 kB] 2021-02-11T09:35:33.727 INFO:teuthology.orchestra.run.smithi133.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [31.8 kB] 2021-02-11T09:35:33.727 INFO:teuthology.orchestra.run.smithi133.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [14.3 kB] 2021-02-11T09:35:33.727 INFO:teuthology.orchestra.run.smithi133.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [7,300 B] 2021-02-11T09:35:34.785 INFO:teuthology.orchestra.run.smithi133.stdout:Fetched 13.1 MB in 2s (6,332 kB/s) 2021-02-11T09:35:36.220 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2021-02-11T09:35:36.265 DEBUG:teuthology.orchestra.run.smithi133:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-11T09:35:36.341 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2021-02-11T09:35:36.572 INFO:teuthology.orchestra.run.smithi133.stdout:Building dependency tree... 2021-02-11T09:35:36.573 INFO:teuthology.orchestra.run.smithi133.stdout:Reading state information... 2021-02-11T09:35:36.652 INFO:teuthology.orchestra.run.smithi133.stderr:E: Unable to locate package linux-image-current-generic 2021-02-11T09:35:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-11T09:35:36.654 DEBUG:teuthology.orchestra.run.smithi133:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-11T09:35:36.775 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2021-02-11T09:35:37.035 INFO:teuthology.orchestra.run.smithi133.stdout:Building dependency tree... 2021-02-11T09:35:37.035 INFO:teuthology.orchestra.run.smithi133.stdout:Reading state information... 2021-02-11T09:35:37.213 INFO:teuthology.orchestra.run.smithi133.stdout:The following additional packages will be installed: 2021-02-11T09:35:37.213 INFO:teuthology.orchestra.run.smithi133.stdout: linux-generic linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-11T09:35:37.214 INFO:teuthology.orchestra.run.smithi133.stdout: linux-headers-generic linux-image-4.15.0-135-generic 2021-02-11T09:35:37.214 INFO:teuthology.orchestra.run.smithi133.stdout: linux-modules-4.15.0-135-generic linux-modules-extra-4.15.0-135-generic 2021-02-11T09:35:37.216 INFO:teuthology.orchestra.run.smithi133.stdout:Suggested packages: 2021-02-11T09:35:37.216 INFO:teuthology.orchestra.run.smithi133.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2021-02-11T09:35:37.216 INFO:teuthology.orchestra.run.smithi133.stdout:Recommended packages: 2021-02-11T09:35:37.216 INFO:teuthology.orchestra.run.smithi133.stdout: thermald 2021-02-11T09:35:37.245 INFO:teuthology.orchestra.run.smithi133.stdout:The following NEW packages will be installed: 2021-02-11T09:35:37.245 INFO:teuthology.orchestra.run.smithi133.stdout: linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-11T09:35:37.245 INFO:teuthology.orchestra.run.smithi133.stdout: linux-image-4.15.0-135-generic linux-modules-4.15.0-135-generic 2021-02-11T09:35:37.246 INFO:teuthology.orchestra.run.smithi133.stdout: linux-modules-extra-4.15.0-135-generic 2021-02-11T09:35:37.247 INFO:teuthology.orchestra.run.smithi133.stdout:The following packages will be upgraded: 2021-02-11T09:35:37.248 INFO:teuthology.orchestra.run.smithi133.stdout: linux-generic linux-headers-generic linux-image-generic 2021-02-11T09:35:37.423 INFO:teuthology.orchestra.run.smithi133.stdout:3 upgraded, 5 newly installed, 0 to remove and 23 not upgraded. 2021-02-11T09:35:37.423 INFO:teuthology.orchestra.run.smithi133.stdout:Need to get 66.0 MB of archives. 2021-02-11T09:35:37.423 INFO:teuthology.orchestra.run.smithi133.stdout:After this operation, 334 MB of additional disk space will be used. 2021-02-11T09:35:37.423 INFO:teuthology.orchestra.run.smithi133.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-135-generic amd64 4.15.0-135.139 [13.1 MB] 2021-02-11T09:35:38.430 INFO:teuthology.orchestra.run.smithi133.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-135-generic amd64 4.15.0-135.139 [8,019 kB] 2021-02-11T09:35:38.629 INFO:teuthology.orchestra.run.smithi133.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-135-generic amd64 4.15.0-135.139 [32.8 MB] 2021-02-11T09:35:39.593 INFO:teuthology.orchestra.run.smithi133.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.135.122 [1,868 B] 2021-02-11T09:35:39.593 INFO:teuthology.orchestra.run.smithi133.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.135.122 [2,592 B] 2021-02-11T09:35:39.593 INFO:teuthology.orchestra.run.smithi133.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135 all 4.15.0-135.139 [10.9 MB] 2021-02-11T09:35:39.983 INFO:teuthology.orchestra.run.smithi133.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135-generic amd64 4.15.0-135.139 [1,097 kB] 2021-02-11T09:35:40.049 INFO:teuthology.orchestra.run.smithi133.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.135.122 [2,488 B] 2021-02-11T09:35:40.534 INFO:teuthology.orchestra.run.smithi133.stdout:Fetched 66.0 MB in 3s (23.7 MB/s) 2021-02-11T09:35:40.657 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package linux-modules-4.15.0-135-generic. 2021-02-11T09:35:40.691 INFO:teuthology.orchestra.run.smithi133.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102079 files and directories currently installed.) 2021-02-11T09:35:40.694 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../0-linux-modules-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T09:35:40.737 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:35:42.337 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package linux-image-4.15.0-135-generic. 2021-02-11T09:35:42.348 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../1-linux-image-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T09:35:42.468 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:35:42.948 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-135-generic. 2021-02-11T09:35:42.961 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T09:35:42.998 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:35:46.801 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../3-linux-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T09:35:46.918 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T09:35:47.226 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T09:35:47.572 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-image-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T09:35:47.916 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package linux-headers-4.15.0-135. 2021-02-11T09:35:47.929 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../5-linux-headers-4.15.0-135_4.15.0-135.139_all.deb ... 2021-02-11T09:35:47.966 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-11T09:35:52.503 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package linux-headers-4.15.0-135-generic. 2021-02-11T09:35:52.524 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../6-linux-headers-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T09:35:52.562 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:35:54.943 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T09:35:55.067 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-headers-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T09:35:55.416 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:35:55.965 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-11T09:35:56.091 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:35:56.567 INFO:teuthology.orchestra.run.smithi133.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-135-generic 2021-02-11T09:35:56.568 INFO:teuthology.orchestra.run.smithi133.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-135-generic 2021-02-11T09:35:56.692 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:35:57.169 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:35:57.295 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-image-generic (4.15.0.135.122) ... 2021-02-11T09:35:57.421 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-headers-generic (4.15.0.135.122) ... 2021-02-11T09:35:57.584 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-generic (4.15.0.135.122) ... 2021-02-11T09:35:57.782 INFO:teuthology.orchestra.run.smithi133.stdout:Processing triggers for linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:35:57.973 INFO:teuthology.orchestra.run.smithi133.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-02-11T09:35:57.973 INFO:teuthology.orchestra.run.smithi133.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-135-generic 2021-02-11T09:36:12.780 INFO:teuthology.orchestra.run.smithi133.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-02-11T09:36:12.781 INFO:teuthology.orchestra.run.smithi133.stdout:Sourcing file `/etc/default/grub' 2021-02-11T09:36:12.813 INFO:teuthology.orchestra.run.smithi133.stdout:Generating grub configuration file ... 2021-02-11T09:36:13.211 INFO:teuthology.orchestra.run.smithi133.stdout:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-11T09:36:13.254 INFO:teuthology.orchestra.run.smithi133.stdout:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-11T09:36:13.678 INFO:teuthology.orchestra.run.smithi133.stdout:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-11T09:36:13.701 INFO:teuthology.orchestra.run.smithi133.stdout:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-11T09:36:15.198 INFO:teuthology.orchestra.run.smithi133.stdout:done 2021-02-11T09:36:15.605 DEBUG:teuthology.orchestra.run.smithi133:> dpkg -s linux-image-generic 2021-02-11T09:36:15.628 INFO:teuthology.orchestra.run.smithi133.stdout:Package: linux-image-generic 2021-02-11T09:36:15.628 INFO:teuthology.orchestra.run.smithi133.stdout:Status: install ok installed 2021-02-11T09:36:15.628 INFO:teuthology.orchestra.run.smithi133.stdout:Priority: optional 2021-02-11T09:36:15.628 INFO:teuthology.orchestra.run.smithi133.stdout:Section: kernel 2021-02-11T09:36:15.628 INFO:teuthology.orchestra.run.smithi133.stdout:Installed-Size: 17 2021-02-11T09:36:15.628 INFO:teuthology.orchestra.run.smithi133.stdout:Maintainer: Ubuntu Kernel Team 2021-02-11T09:36:15.628 INFO:teuthology.orchestra.run.smithi133.stdout:Architecture: amd64 2021-02-11T09:36:15.628 INFO:teuthology.orchestra.run.smithi133.stdout:Source: linux-meta 2021-02-11T09:36:15.628 INFO:teuthology.orchestra.run.smithi133.stdout:Version: 4.15.0.135.122 2021-02-11T09:36:15.628 INFO:teuthology.orchestra.run.smithi133.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-11T09:36:15.629 INFO:teuthology.orchestra.run.smithi133.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-11T09:36:15.629 INFO:teuthology.orchestra.run.smithi133.stdout:Recommends: thermald 2021-02-11T09:36:15.629 INFO:teuthology.orchestra.run.smithi133.stdout:Description: Generic Linux kernel image 2021-02-11T09:36:15.629 INFO:teuthology.orchestra.run.smithi133.stdout: This package will always depend on the latest generic kernel image 2021-02-11T09:36:15.629 INFO:teuthology.orchestra.run.smithi133.stdout: available. 2021-02-11T09:36:15.629 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-134-generic Expected: 4.15.0-135-generic 2021-02-11T09:36:15.629 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-02-11T09:36:15.629 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-02-11T09:36:15.629 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-02-11T09:36:15.630 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2021-02-11T09:36:15.630 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi026.front.sepia.ceph.com, path=None, version=distro) 2021-02-11T09:36:15.630 DEBUG:teuthology.orchestra.run.smithi026:> sudo apt-get clean 2021-02-11T09:36:15.680 DEBUG:teuthology.orchestra.run.smithi026:> sudo apt-get update 2021-02-11T09:36:15.926 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-02-11T09:36:15.926 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T09:36:15.996 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-02-11T09:36:16.076 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-02-11T09:36:17.873 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2021-02-11T09:36:17.922 DEBUG:teuthology.orchestra.run.smithi026:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-11T09:36:17.999 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2021-02-11T09:36:18.251 INFO:teuthology.orchestra.run.smithi026.stdout:Building dependency tree... 2021-02-11T09:36:18.252 INFO:teuthology.orchestra.run.smithi026.stdout:Reading state information... 2021-02-11T09:36:18.338 INFO:teuthology.orchestra.run.smithi026.stderr:E: Unable to locate package linux-image-current-generic 2021-02-11T09:36:18.340 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-11T09:36:18.340 DEBUG:teuthology.orchestra.run.smithi026:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-11T09:36:18.412 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2021-02-11T09:36:18.661 INFO:teuthology.orchestra.run.smithi026.stdout:Building dependency tree... 2021-02-11T09:36:18.661 INFO:teuthology.orchestra.run.smithi026.stdout:Reading state information... 2021-02-11T09:36:18.873 INFO:teuthology.orchestra.run.smithi026.stdout:linux-image-generic is already the newest version (4.15.0.135.122). 2021-02-11T09:36:18.873 INFO:teuthology.orchestra.run.smithi026.stdout:0 upgraded, 0 newly installed, 0 to remove and 23 not upgraded. 2021-02-11T09:36:18.875 DEBUG:teuthology.orchestra.run.smithi026:> dpkg -s linux-image-generic 2021-02-11T09:36:18.890 INFO:teuthology.orchestra.run.smithi026.stdout:Package: linux-image-generic 2021-02-11T09:36:18.890 INFO:teuthology.orchestra.run.smithi026.stdout:Status: install ok installed 2021-02-11T09:36:18.890 INFO:teuthology.orchestra.run.smithi026.stdout:Priority: optional 2021-02-11T09:36:18.890 INFO:teuthology.orchestra.run.smithi026.stdout:Section: kernel 2021-02-11T09:36:18.891 INFO:teuthology.orchestra.run.smithi026.stdout:Installed-Size: 17 2021-02-11T09:36:18.891 INFO:teuthology.orchestra.run.smithi026.stdout:Maintainer: Ubuntu Kernel Team 2021-02-11T09:36:18.891 INFO:teuthology.orchestra.run.smithi026.stdout:Architecture: amd64 2021-02-11T09:36:18.891 INFO:teuthology.orchestra.run.smithi026.stdout:Source: linux-meta 2021-02-11T09:36:18.891 INFO:teuthology.orchestra.run.smithi026.stdout:Version: 4.15.0.135.122 2021-02-11T09:36:18.891 INFO:teuthology.orchestra.run.smithi026.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-11T09:36:18.891 INFO:teuthology.orchestra.run.smithi026.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-11T09:36:18.891 INFO:teuthology.orchestra.run.smithi026.stdout:Recommends: thermald 2021-02-11T09:36:18.891 INFO:teuthology.orchestra.run.smithi026.stdout:Description: Generic Linux kernel image 2021-02-11T09:36:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: This package will always depend on the latest generic kernel image 2021-02-11T09:36:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: available. 2021-02-11T09:36:18.892 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2021-02-11T09:36:18.936 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.Yv4mbNx10c 2021-02-11T09:36:18.937 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Yv4mbNx10c 2021-02-11T09:36:18.992 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.Yv4mbNx10c 2021-02-11T09:36:19.109 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.Yv4mbNx10c is 8KB 2021-02-11T09:36:19.161 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.Yv4mbNx10c 2021-02-11T09:36:19.167 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-02-11T09:36:19.221 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:36:19.221 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-02-11T09:36:19.221 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-02-11T09:36:19.294 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-135-generic 2021-02-11T09:36:19.294 DEBUG:teuthology.orchestra.run.smithi026:> sudo update-grub 2021-02-11T09:36:20.609 INFO:teuthology.orchestra.run.smithi026.stderr:Sourcing file `/etc/default/grub' 2021-02-11T09:36:20.628 INFO:teuthology.orchestra.run.smithi026.stderr:Generating grub configuration file ... 2021-02-11T09:36:20.912 INFO:teuthology.orchestra.run.smithi026.stderr:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-11T09:36:20.925 INFO:teuthology.orchestra.run.smithi026.stderr:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-11T09:36:21.323 INFO:teuthology.orchestra.run.smithi026.stderr:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-11T09:36:21.336 INFO:teuthology.orchestra.run.smithi026.stderr:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-11T09:36:22.565 INFO:teuthology.orchestra.run.smithi026.stderr:done 2021-02-11T09:36:22.566 DEBUG:teuthology.orchestra.run.smithi026:> sudo shutdown -r now 2021-02-11T09:36:22.568 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2021-02-11T09:36:22.568 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi133.front.sepia.ceph.com, path=None, version=distro) 2021-02-11T09:36:22.568 DEBUG:teuthology.orchestra.run.smithi133:> sudo apt-get clean 2021-02-11T09:36:22.616 DEBUG:teuthology.orchestra.run.smithi133:> sudo apt-get update 2021-02-11T09:36:22.752 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-02-11T09:36:22.857 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T09:36:22.937 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-02-11T09:36:23.016 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-02-11T09:36:24.622 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2021-02-11T09:36:24.666 DEBUG:teuthology.orchestra.run.smithi133:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-11T09:36:24.744 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2021-02-11T09:36:25.002 INFO:teuthology.orchestra.run.smithi133.stdout:Building dependency tree... 2021-02-11T09:36:25.003 INFO:teuthology.orchestra.run.smithi133.stdout:Reading state information... 2021-02-11T09:36:25.085 INFO:teuthology.orchestra.run.smithi133.stderr:E: Unable to locate package linux-image-current-generic 2021-02-11T09:36:25.086 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-11T09:36:25.087 DEBUG:teuthology.orchestra.run.smithi133:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-11T09:36:25.164 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2021-02-11T09:36:25.394 INFO:teuthology.orchestra.run.smithi133.stdout:Building dependency tree... 2021-02-11T09:36:25.395 INFO:teuthology.orchestra.run.smithi133.stdout:Reading state information... 2021-02-11T09:36:25.594 INFO:teuthology.orchestra.run.smithi133.stdout:linux-image-generic is already the newest version (4.15.0.135.122). 2021-02-11T09:36:25.594 INFO:teuthology.orchestra.run.smithi133.stdout:0 upgraded, 0 newly installed, 0 to remove and 23 not upgraded. 2021-02-11T09:36:25.596 DEBUG:teuthology.orchestra.run.smithi133:> dpkg -s linux-image-generic 2021-02-11T09:36:25.616 INFO:teuthology.orchestra.run.smithi133.stdout:Package: linux-image-generic 2021-02-11T09:36:25.616 INFO:teuthology.orchestra.run.smithi133.stdout:Status: install ok installed 2021-02-11T09:36:25.616 INFO:teuthology.orchestra.run.smithi133.stdout:Priority: optional 2021-02-11T09:36:25.616 INFO:teuthology.orchestra.run.smithi133.stdout:Section: kernel 2021-02-11T09:36:25.616 INFO:teuthology.orchestra.run.smithi133.stdout:Installed-Size: 17 2021-02-11T09:36:25.616 INFO:teuthology.orchestra.run.smithi133.stdout:Maintainer: Ubuntu Kernel Team 2021-02-11T09:36:25.616 INFO:teuthology.orchestra.run.smithi133.stdout:Architecture: amd64 2021-02-11T09:36:25.616 INFO:teuthology.orchestra.run.smithi133.stdout:Source: linux-meta 2021-02-11T09:36:25.616 INFO:teuthology.orchestra.run.smithi133.stdout:Version: 4.15.0.135.122 2021-02-11T09:36:25.617 INFO:teuthology.orchestra.run.smithi133.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-11T09:36:25.617 INFO:teuthology.orchestra.run.smithi133.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-11T09:36:25.617 INFO:teuthology.orchestra.run.smithi133.stdout:Recommends: thermald 2021-02-11T09:36:25.617 INFO:teuthology.orchestra.run.smithi133.stdout:Description: Generic Linux kernel image 2021-02-11T09:36:25.617 INFO:teuthology.orchestra.run.smithi133.stdout: This package will always depend on the latest generic kernel image 2021-02-11T09:36:25.617 INFO:teuthology.orchestra.run.smithi133.stdout: available. 2021-02-11T09:36:25.617 DEBUG:teuthology.orchestra.run.smithi133:> mktemp 2021-02-11T09:36:25.664 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.e78oZnlpId 2021-02-11T09:36:25.664 DEBUG:teuthology.orchestra.run.smithi133:> sudo cp /boot/grub/grub.cfg /tmp/tmp.e78oZnlpId 2021-02-11T09:36:25.720 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.e78oZnlpId 2021-02-11T09:36:25.833 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.e78oZnlpId is 8KB 2021-02-11T09:36:25.841 DEBUG:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.e78oZnlpId 2021-02-11T09:36:25.847 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-02-11T09:36:25.908 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:36:25.908 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-02-11T09:36:25.908 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-02-11T09:36:25.979 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-135-generic 2021-02-11T09:36:25.979 DEBUG:teuthology.orchestra.run.smithi133:> sudo update-grub 2021-02-11T09:36:27.422 INFO:teuthology.orchestra.run.smithi133.stderr:Sourcing file `/etc/default/grub' 2021-02-11T09:36:27.444 INFO:teuthology.orchestra.run.smithi133.stderr:Generating grub configuration file ... 2021-02-11T09:36:27.697 INFO:teuthology.orchestra.run.smithi133.stderr:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-11T09:36:27.709 INFO:teuthology.orchestra.run.smithi133.stderr:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-11T09:36:28.044 INFO:teuthology.orchestra.run.smithi133.stderr:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-11T09:36:28.057 INFO:teuthology.orchestra.run.smithi133.stderr:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-11T09:36:29.205 INFO:teuthology.orchestra.run.smithi133.stderr:done 2021-02-11T09:36:29.207 DEBUG:teuthology.orchestra.run.smithi133:> sudo shutdown -r now 2021-02-11T09:36:59.227 INFO:teuthology.misc:Re-opening connections... 2021-02-11T09:36:59.227 INFO:teuthology.misc:trying to connect to ubuntu@smithi026.front.sepia.ceph.com 2021-02-11T09:36:59.228 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T09:36:59.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T09:37:59.231 DEBUG:teuthology.orchestra.remote:timed out 2021-02-11T09:37:59.231 INFO:teuthology.misc:trying to connect to ubuntu@smithi133.front.sepia.ceph.com 2021-02-11T09:37:59.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T09:37:59.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T09:38:14.677 DEBUG:teuthology.orchestra.run.smithi133:> true 2021-02-11T09:38:15.129 DEBUG:teuthology.misc:waited 75.9011869430542 2021-02-11T09:38:16.129 INFO:teuthology.misc:trying to connect to ubuntu@smithi026.front.sepia.ceph.com 2021-02-11T09:38:16.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T09:38:16.130 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T09:38:16.211 DEBUG:teuthology.orchestra.run.smithi026:> true 2021-02-11T09:38:16.688 DEBUG:teuthology.misc:waited 77.46082043647766 2021-02-11T09:38:17.689 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2021-02-11T09:38:17.689 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.15.0-135-generic"... 2021-02-11T09:38:17.689 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2021-02-11T09:38:17.695 INFO:teuthology.orchestra.run.smithi026.stdout:4.15.0-135-generic 2021-02-11T09:38:17.695 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-135-generic vs 4.15.0-135-generic 2021-02-11T09:38:17.696 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-02-11T09:38:17.696 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-02-11T09:38:17.696 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2021-02-11T09:38:17.696 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.15.0-135-generic"... 2021-02-11T09:38:17.696 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2021-02-11T09:38:17.702 INFO:teuthology.orchestra.run.smithi133.stdout:4.15.0-135-generic 2021-02-11T09:38:17.702 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-135-generic vs 4.15.0-135-generic 2021-02-11T09:38:17.702 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-02-11T09:38:17.702 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-02-11T09:38:18.703 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2021-02-11T09:38:18.704 DEBUG:teuthology.orchestra.run.smithi026:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-11T09:38:18.730 INFO:teuthology.orchestra.run.smithi026.stdout:ttyS1 2021-02-11T09:38:18.764 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2021-02-11T09:38:18.764 DEBUG:teuthology.orchestra.run.smithi133:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-11T09:38:18.790 INFO:teuthology.orchestra.run.smithi133.stdout:ttyS1 2021-02-11T09:38:18.831 INFO:teuthology.run_tasks:Running task internal.base... 2021-02-11T09:38:18.838 INFO:teuthology.task.internal:Creating test directory... 2021-02-11T09:38:18.838 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-11T09:38:18.841 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-11T09:38:18.879 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-02-11T09:38:18.886 INFO:teuthology.run_tasks:Running task internal.archive... 2021-02-11T09:38:18.903 INFO:teuthology.task.internal:Creating archive directory... 2021-02-11T09:38:18.904 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-11T09:38:18.906 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-11T09:38:18.941 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-02-11T09:38:18.951 INFO:teuthology.task.internal:Enabling coredump saving... 2021-02-11T09:38:18.951 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-11T09:38:18.959 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-11T09:38:18.987 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-11T09:38:18.996 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-11T09:38:19.001 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-11T09:38:19.008 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-11T09:38:19.009 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-02-11T09:38:19.020 INFO:teuthology.task.internal:Configuring sudo... 2021-02-11T09:38:19.020 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-11T09:38:19.043 DEBUG:teuthology.orchestra.run.smithi133:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-11T09:38:19.068 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-02-11T09:38:19.091 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-02-11T09:38:19.091 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-11T09:38:19.099 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-11T09:38:19.116 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-11T09:38:19.152 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-11T09:38:19.200 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:38:19.200 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-11T09:38:19.266 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-11T09:38:19.272 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-11T09:38:19.319 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:38:19.319 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-11T09:38:19.384 DEBUG:teuthology.orchestra.run.smithi026:> sudo service rsyslog restart 2021-02-11T09:38:19.386 DEBUG:teuthology.orchestra.run.smithi133:> sudo service rsyslog restart 2021-02-11T09:38:19.465 INFO:teuthology.run_tasks:Running task internal.timer... 2021-02-11T09:38:19.472 INFO:teuthology.task.internal:Starting timer... 2021-02-11T09:38:19.472 INFO:teuthology.run_tasks:Running task pcp... 2021-02-11T09:38:19.495 INFO:teuthology.run_tasks:Running task selinux... 2021-02-11T09:38:19.510 DEBUG:teuthology.task.selinux:Excluding smithi026: OS 'ubuntu' does not support SELinux 2021-02-11T09:38:19.510 DEBUG:teuthology.task.selinux:Excluding smithi133: OS 'ubuntu' does not support SELinux 2021-02-11T09:38:19.511 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-02-11T09:38:19.511 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2021-02-11T09:38:19.511 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-02-11T09:38:19.511 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-02-11T09:38:19.528 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-02-11T09:38:19.528 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-02-11T09:38:19.730 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-02-11T09:38:19.737 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi026.front.sepia.ceph.com,smithi133.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-02-11T09:38:20.274 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-02-11T09:38:21.852 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-02-11T09:38:24.543 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-02-11T09:38:24.576 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:24 +0000 (0:00:00.141) 0:00:00.141 ***** 2021-02-11T09:38:24.593 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:24 +0000 (0:00:00.017) 0:00:00.158 ***** 2021-02-11T09:38:26.374 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi133.front.sepia.ceph.com] 2021-02-11T09:38:26.420 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] 2021-02-11T09:38:26.484 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:01.890) 0:00:02.049 ***** 2021-02-11T09:38:26.526 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.041) 0:00:02.091 ***** 2021-02-11T09:38:26.742 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-02-11T09:38:26.743 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-02-11T09:38:26.778 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.252) 0:00:02.343 ***** 2021-02-11T09:38:26.813 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.034) 0:00:02.378 ***** 2021-02-11T09:38:27.224 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-02-11T09:38:27.225 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-02-11T09:38:27.261 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.448) 0:00:02.827 ***** 2021-02-11T09:38:27.269 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-02-11T09:38:27.307 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.045) 0:00:02.872 ***** 2021-02-11T09:38:27.595 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-02-11T09:38:27.596 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.075684", "end": "2021-02-11 09:38:27.944354", "rc": 0, "start": "2021-02-11 09:38:27.868670", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-02-11T09:38:27.629 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.321) 0:00:03.194 ***** 2021-02-11T09:38:27.635 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.097625", "end": "2021-02-11 09:38:27.579012", "rc": 0, "start": "2021-02-11 09:38:27.481387", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-02-11T09:38:27.672 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.043) 0:00:03.237 ***** 2021-02-11T09:38:27.873 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-02-11T09:38:27.875 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi026.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T09:38:27.910 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.238) 0:00:03.475 ***** 2021-02-11T09:38:27.943 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.032) 0:00:03.508 ***** 2021-02-11T09:38:28.649 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-02-11T09:38:28.651 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036307.9565299-208510039586278/source", "state": "file", "uid": 0} changed: [smithi026.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036307.9226964-57733590493266/source", "state": "file", "uid": 0} 2021-02-11T09:38:28.688 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:28 +0000 (0:00:00.744) 0:00:04.253 ***** 2021-02-11T09:38:28.722 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:28 +0000 (0:00:00.034) 0:00:04.287 ***** 2021-02-11T09:38:28.986 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-02-11T09:38:28.988 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-02-11T09:38:28.989 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-02-11T09:38:29.023 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-02-11T09:38:29.024 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-02-11T09:38:29.138 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-02-11T09:38:29.145 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-02-11T09:38:29.291 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:29 +0000 (0:00:00.569) 0:00:04.857 ***** 2021-02-11T09:38:29.325 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:29 +0000 (0:00:00.033) 0:00:04.890 ***** 2021-02-11T09:38:30.046 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi133.front.sepia.ceph.com] 2021-02-11T09:38:30.056 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] 2021-02-11T09:38:30.157 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-02-11T09:38:30.159 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:30 +0000 (0:00:00.832) 0:00:05.722 ***** 2021-02-11T09:38:30.195 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:30 +0000 (0:00:00.037) 0:00:05.760 ***** 2021-02-11T09:38:30.954 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-02-11T09:38:30.955 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.429582", "end": "2021-02-11 09:38:30.925858", "rc": 0, "start": "2021-02-11 09:38:30.496276", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-02-11T09:38:30.955 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.420982", "end": "2021-02-11 09:38:30.918131", "rc": 0, "start": "2021-02-11 09:38:30.497149", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-02-11T09:38:30.990 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:30 +0000 (0:00:00.794) 0:00:06.555 ***** 2021-02-11T09:38:31.023 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.033) 0:00:06.588 ***** 2021-02-11T09:38:31.124 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-02-11T09:38:31.125 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.013599", "end": "2021-02-11 09:38:31.110665", "failed_when_result": false, "rc": 0, "start": "2021-02-11 09:38:31.097066", "stderr": "", "stderr_lines": [], "stdout": "0caff10a478f2674721da1965013b225f3ea5800", "stdout_lines": ["0caff10a478f2674721da1965013b225f3ea5800"]} 2021-02-11T09:38:31.162 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.138) 0:00:06.727 ***** 2021-02-11T09:38:31.189 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2021-02-11T09:38:31.191 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:31.225 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.062) 0:00:06.790 ***** 2021-02-11T09:38:31.231 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-02-11T09:38:31.231 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.066589", "end": "2021-02-11 09:38:31.561083", "failed_when_result": false, "rc": 0, "start": "2021-02-11 09:38:31.494494", "stderr": "", "stderr_lines": [], "stdout": "0caff10a478f2674721da1965013b225f3ea5800", "stdout_lines": ["0caff10a478f2674721da1965013b225f3ea5800"]} 2021-02-11T09:38:31.267 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.042) 0:00:06.832 ***** 2021-02-11T09:38:34.417 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:38:31 +0000 (0:00:00.041) 0:00:06.873 ***** TASK [users : Determine if we can skip users and pubkeys updates] ************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:38:31 +0000 (0:00:00.042) 0:00:06.916 ***** TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:38:31 +0000 (0:00:00.041) 0:00:06.958 ***** TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:38:31 +0000 (0:00:00.042) 0:00:07.000 ***** TASK [users : Filter the managed_users list] *********************************** skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:38:31 +0000 (0:00:00.041) 0:00:07.042 ***** TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:38:31 +0000 (0:00:00.042) 0:00:07.084 ***** TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:38:31 +0000 (0:00:00.044) 0:00:07.129 ***** TASK [users : Filter the managed_users list] 2021-02-11T09:38:34.419 INFO:teuthology.task.ansible.out:*********************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:38:31 +0000 (0:00:00.042) 0:00:07.171 ***** TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:38:31 +0000 (0:00:00.060) 0:00:07.232 ***** TASK [users : Set UID_MIN to 1001] ********************************************* ok: [smithi026.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Thursday 11 February 2021 09:38:31 +0000 (0:00:00.054) 0:00:07.286 ***** ok: [smithi133.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Thursday 11 February 2021 09:38:31 +0000 (0:00:00.126) 0:00:07.412 ***** TASK [users : Create all admin users with sudo access.] ************************ ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8 2021-02-11T09:38:34.420 INFO:teuthology.task.ansible.out:t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdrey 2021-02-11T09:38:34.421 INFO:teuthology.task.ansible.out:er", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hp 2021-02-11T09:38:34.423 INFO:teuthology.task.ansible.out:laptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e87 2021-02-11T09:38:34.424 INFO:teuthology.task.ansible.out:4ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3 2021-02-11T09:38:34.425 INFO:teuthology.task.ansible.out:332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} ok: [smit 2021-02-11T09:38:34.427 INFO:teuthology.task.ansible.out:hi133.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/hom 2021-02-11T09:38:34.428 INFO:teuthology.task.ansible.out:e/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", 2021-02-11T09:38:34.428 INFO:teuthology.task.ansible.out:"shell": "/bin/bash", "state": "present", "uid": 1108} 2021-02-11T09:38:34.430 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-11T09:38:34.432 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-02-11T09:38:34.434 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-11T09:38:34.436 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-11T09:38:34.438 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-11T09:38:34.439 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-11T09:38:34.440 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-02-11T09:38:34.442 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-11T09:38:34.443 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-02-11T09:38:34.444 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-02-11T09:38:34.445 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-02-11T09:38:34.446 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-02-11T09:38:34.447 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-02-11T09:38:34.449 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-02-11T09:38:34.449 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-02-11T09:38:34.450 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-02-11T09:38:34.451 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-02-11T09:38:34.453 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-02-11T09:38:34.454 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-02-11T09:38:34.454 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-02-11T09:38:34.455 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-02-11T09:38:34.467 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-02-11T09:38:34.491 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-02-11T09:38:34.587 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-02-11T09:38:34.619 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-02-11T09:38:34.714 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-11T09:38:34.768 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-02-11T09:38:34.842 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-11T09:38:34.905 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-02-11T09:38:34.960 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-02-11T09:38:35.034 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-11T09:38:35.098 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-11T09:38:35.172 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-11T09:38:35.225 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-11T09:38:35.319 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-02-11T09:38:35.351 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-11T09:38:35.445 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-11T09:38:35.477 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-11T09:38:35.582 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-11T09:38:35.603 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-02-11T09:38:35.718 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-11T09:38:35.730 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-02-11T09:38:35.856 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-02-11T09:38:35.857 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-11T09:38:35.983 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-11T09:38:35.994 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-02-11T09:38:36.109 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-11T09:38:36.142 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-02-11T09:38:36.236 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-11T09:38:36.259 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-02-11T09:38:36.354 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-11T09:38:36.386 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-11T09:38:36.491 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-11T09:38:36.534 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-11T09:38:36.617 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-02-11T09:38:36.660 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-11T09:38:36.743 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-11T09:38:36.796 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-11T09:38:36.869 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-11T09:38:36.946 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-11T09:38:36.999 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-02-11T09:38:37.083 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-02-11T09:38:37.125 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-11T09:38:37.209 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-11T09:38:37.258 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-02-11T09:38:37.352 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-11T09:38:37.384 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-11T09:38:37.489 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-02-11T09:38:37.511 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-02-11T09:38:37.616 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-11T09:38:37.638 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-11T09:38:37.732 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-02-11T09:38:37.764 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-11T09:38:37.880 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-11T09:38:37.891 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-11T09:38:38.016 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-11T09:38:38.028 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-02-11T09:38:38.143 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-02-11T09:38:38.154 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-11T09:38:38.269 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-11T09:38:38.290 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-11T09:38:38.395 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-02-11T09:38:38.417 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-11T09:38:38.522 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-11T09:38:38.554 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-11T09:38:38.658 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-11T09:38:38.701 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-02-11T09:38:38.784 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-11T09:38:38.827 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-11T09:38:38.911 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-11T09:38:38.953 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-02-11T09:38:39.047 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-11T09:38:39.079 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-11T09:38:39.174 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-11T09:38:39.216 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-11T09:38:39.300 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-11T09:38:39.363 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-11T09:38:39.427 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-11T09:38:39.490 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-11T09:38:39.554 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-11T09:38:39.628 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-11T09:38:39.681 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-11T09:38:39.744 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-11T09:38:39.808 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-11T09:38:39.860 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-11T09:38:39.934 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-11T09:38:40.008 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-11T09:38:40.092 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-11T09:38:40.145 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-11T09:38:40.219 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-11T09:38:40.272 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-11T09:38:40.346 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-11T09:38:40.399 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-11T09:38:40.474 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-11T09:38:40.548 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-11T09:38:40.602 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-11T09:38:40.675 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-11T09:38:40.728 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-11T09:38:40.802 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-11T09:38:40.844 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-11T09:38:40.938 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-11T09:38:40.981 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-11T09:38:41.075 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-11T09:38:41.108 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-02-11T09:38:41.203 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-11T09:38:41.228 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-11T09:38:41.343 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-11T09:38:41.364 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-11T09:38:41.480 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-11T09:38:41.492 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-11T09:38:41.607 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-11T09:38:41.629 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-11T09:38:41.744 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-02-11T09:38:41.765 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-02-11T09:38:41.870 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-02-11T09:38:41.913 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-11T09:38:42.007 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-11T09:38:42.059 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-11T09:38:42.133 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-11T09:38:42.196 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-11T09:38:42.260 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-11T09:38:42.323 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-11T09:38:42.386 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-02-11T09:38:42.460 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-02-11T09:38:42.523 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-02-11T09:38:42.597 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-02-11T09:38:42.662 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-02-11T09:38:42.726 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-11T09:38:42.779 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-02-11T09:38:42.873 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-11T09:38:42.905 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-11T09:38:43.000 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-11T09:38:43.032 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-11T09:38:43.137 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-02-11T09:38:43.159 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-11T09:38:43.284 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-02-11T09:38:43.285 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-02-11T09:38:43.411 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-02-11T09:38:43.412 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-02-11T09:38:43.538 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-02-11T09:38:43.560 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-02-11T09:38:43.665 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-02-11T09:38:43.697 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-11T09:38:43.792 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-02-11T09:38:43.835 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-11T09:38:43.919 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-02-11T09:38:44.003 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-11T09:38:44.045 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-02-11T09:38:44.152 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-02-11T09:38:44.185 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-02-11T09:38:44.290 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-02-11T09:38:44.301 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-02-11T09:38:44.417 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-02-11T09:38:44.428 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-02-11T09:38:44.554 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-02-11T09:38:44.555 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-02-11T09:38:44.681 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-02-11T09:38:44.692 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-02-11T09:38:44.807 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-11T09:38:44.829 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-02-11T09:38:44.934 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-02-11T09:38:44.966 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-02-11T09:38:45.062 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-02-11T09:38:45.116 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-02-11T09:38:45.201 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-02-11T09:38:45.254 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-02-11T09:38:45.318 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-02-11T09:38:45.392 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-02-11T09:38:45.445 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-02-11T09:38:45.530 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-02-11T09:38:45.573 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-02-11T09:38:45.678 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-02-11T09:38:45.700 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-02-11T09:38:45.825 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-02-11T09:38:45.826 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-11T09:38:45.952 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-02-11T09:38:45.974 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-02-11T09:38:46.081 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-02-11T09:38:46.103 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-02-11T09:38:46.208 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-02-11T09:38:46.240 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-02-11T09:38:46.335 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-11T09:38:46.378 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-02-11T09:38:46.462 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-11T09:38:46.525 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-02-11T09:38:46.578 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-02-11T09:38:46.652 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-02-11T09:38:46.706 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-11T09:38:46.780 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-02-11T09:38:46.844 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-11T09:38:46.910 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-02-11T09:38:46.974 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-11T09:38:47.050 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-02-11T09:38:47.093 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-02-11T09:38:47.188 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-02-11T09:38:47.219 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-02-11T09:38:47.334 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-02-11T09:38:47.356 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-02-11T09:38:47.460 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-11T09:38:47.482 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-02-11T09:38:47.587 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-11T09:38:47.608 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-02-11T09:38:47.723 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-02-11T09:38:47.735 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-11T09:38:47.860 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-02-11T09:38:47.861 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-11T09:38:47.988 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-02-11T09:38:47.999 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-11T09:38:48.114 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-11T09:38:48.135 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-11T09:38:48.240 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-11T09:38:48.283 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-02-11T09:38:48.366 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-02-11T09:38:48.419 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-02-11T09:38:48.492 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-11T09:38:48.556 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-02-11T09:38:48.619 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-02-11T09:38:48.704 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-02-11T09:38:48.746 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-11T09:38:48.851 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-02-11T09:38:48.873 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-11T09:38:48.987 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-11T09:38:49.010 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-02-11T09:38:49.126 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-02-11T09:38:49.138 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-11T09:38:49.264 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-02-11T09:38:49.276 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-02-11T09:38:49.402 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-02-11T09:38:49.414 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-11T09:38:49.550 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-11T09:38:49.686 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-02-11T09:38:49.831 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-11T09:38:49.906 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-02-11T09:38:49.986 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-02-11T09:38:50.023 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:50 +0000 (0:00:18.175) 0:00:25.588 ***** 2021-02-11T09:38:50.054 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2021-02-11T09:38:50.088 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:50 +0000 (0:00:00.065) 0:00:25.653 ***** 2021-02-11T09:38:50.104 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-11T09:38:50.106 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-11T09:38:50.127 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2021-02-11T09:38:50.129 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-02-11T09:38:50.130 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-02-11T09:38:50.132 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancic 2021-02-11T09:38:50.133 INFO:teuthology.task.ansible.out:h", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1Wi 2021-02-11T09:38:50.134 INFO:teuthology.task.ansible.out:RenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdilla 2021-02-11T09:38:50.135 INFO:teuthology.task.ansible.out:man@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vum 2021-02-11T09:38:50.136 INFO:teuthology.task.ansible.out:rao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69 2021-02-11T09:38:50.137 INFO:teuthology.task.ansible.out:mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d8 2021-02-11T09:38:50.139 INFO:teuthology.task.ansible.out:41a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveirada 2021-02-11T09:38:50.140 INFO:teuthology.task.ansible.out:n@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOe 2021-02-11T09:38:50.141 INFO:teuthology.task.ansible.out:nVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9 2021-02-11T09:38:50.143 INFO:teuthology.task.ansible.out:gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": 2021-02-11T09:38:50.144 INFO:teuthology.task.ansible.out:"dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NI 2021-02-11T09:38:50.145 INFO:teuthology.task.ansible.out:Nj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-02-11T09:38:50.185 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:50 +0000 (0:00:00.097) 0:00:25.750 ***** 2021-02-11T09:38:50.244 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-11T09:38:50.245 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-11T09:38:50.382 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-02-11T09:38:50.517 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-11T09:38:50.632 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-02-11T09:38:50.779 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-02-11T09:38:50.997 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-02-11T09:38:50.999 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com -> localhost] => {"after": "df8946f08e870e2d2565b79099289ca48bd825f3", "attempts": 1, "before": "df8946f08e870e2d2565b79099289ca48bd825f3", "changed": false, "remote_url_changed": false} 2021-02-11T09:38:51.032 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:51 +0000 (0:00:00.846) 0:00:26.597 ***** 2021-02-11T09:38:51.225 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:51.226 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-02-11T09:38:51.340 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-11T09:38:51.341 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-02-11T09:38:51.457 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:51.459 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-02-11T09:38:51.499 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:51 +0000 (0:00:00.466) 0:00:27.064 ***** 2021-02-11T09:38:51.505 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-02-11T09:38:51.520 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2021-02-11T09:38:51.556 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:51 +0000 (0:00:00.057) 0:00:27.122 ***** 2021-02-11T09:38:51.598 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2021-02-11T09:38:51.601 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-02-11T09:38:51.602 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-02-11T09:38:51.604 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancic 2021-02-11T09:38:51.605 INFO:teuthology.task.ansible.out:h", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1Wi 2021-02-11T09:38:51.606 INFO:teuthology.task.ansible.out:RenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovp 2021-02-11T09:38:51.606 INFO:teuthology.task.ansible.out:n": "vumrao@vum 2021-02-11T09:38:51.608 INFO:teuthology.task.ansible.out:rao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69 2021-02-11T09:38:51.609 INFO:teuthology.task.ansible.out:mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d8 2021-02-11T09:38:51.610 INFO:teuthology.task.ansible.out:41a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveirada 2021-02-11T09:38:51.611 INFO:teuthology.task.ansible.out:n@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOe 2021-02-11T09:38:51.613 INFO:teuthology.task.ansible.out:nVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9 2021-02-11T09:38:51.614 INFO:teuthology.task.ansible.out:gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": 2021-02-11T09:38:51.615 INFO:teuthology.task.ansible.out:"dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "ac 2021-02-11T09:38:51.616 INFO:teuthology.task.ansible.out:lewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NI 2021-02-11T09:38:51.617 INFO:teuthology.task.ansible.out:Nj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-02-11T09:38:51.620 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:51.620 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-02-11T09:38:51.621 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-02-11T09:38:51.664 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:51 +0000 (0:00:00.107) 0:00:27.229 ***** 2021-02-11T09:38:51.743 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-11T09:38:51.879 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-02-11T09:38:51.881 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-02-11T09:38:52.005 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-11T09:38:52.131 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-11T09:38:52.206 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-02-11T09:38:52.207 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com -> localhost] => {"after": "df8946f08e870e2d2565b79099289ca48bd825f3", "attempts": 1, "before": "df8946f08e870e2d2565b79099289ca48bd825f3", "changed": false, "remote_url_changed": false} 2021-02-11T09:38:52.241 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:52 +0000 (0:00:00.577) 0:00:27.807 ***** 2021-02-11T09:38:52.259 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:52.261 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1s 2021-02-11T09:38:52.262 INFO:teuthology.task.ansible.out:cr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:52.386 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-02-11T09:38:52.387 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-02-11T09:38:52.418 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-02-11T09:38:52.502 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-02-11T09:38:52.503 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-02-11T09:38:52.513 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:52.524 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:52.566 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-02-11T09:38:52.651 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-11T09:38:52.683 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-02-11T09:38:52.767 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-02-11T09:38:52.769 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-02-11T09:38:52.820 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-02-11T09:38:52.821 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-02-11T09:38:52.893 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-11T09:38:52.966 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-11T09:38:53.019 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-11T09:38:53.115 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-02-11T09:38:53.116 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-02-11T09:38:53.147 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-11T09:38:53.240 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-11T09:38:53.261 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-02-11T09:38:53.366 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-11T09:38:53.388 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-02-11T09:38:53.502 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-02-11T09:38:53.513 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:53.514 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-02-11T09:38:53.515 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-02-11T09:38:53.651 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-02-11T09:38:53.652 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNO 2021-02-11T09:38:53.654 INFO:teuthology.task.ansible.out:VdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-02-11T09:38:53.787 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-11T09:38:53.798 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-02-11T09:38:53.800 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-02-11T09:38:53.809 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:53.810 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:53.914 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-02-11T09:38:53.956 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-11T09:38:54.040 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-02-11T09:38:54.041 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-02-11T09:38:54.072 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-02-11T09:38:54.073 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-02-11T09:38:54.158 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-02-11T09:38:54.200 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-11T09:38:54.284 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-02-11T09:38:54.347 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-11T09:38:54.410 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-02-11T09:38:54.483 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-11T09:38:54.535 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-02-11T09:38:54.619 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-02-11T09:38:54.671 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-02-11T09:38:54.744 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-02-11T09:38:54.797 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-02-11T09:38:54.798 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-02-11T09:38:54.880 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-02-11T09:38:54.882 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-02-11T09:38:54.922 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-02-11T09:38:55.006 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-02-11T09:38:55.008 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-02-11T09:38:55.060 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-02-11T09:38:55.123 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-11T09:38:55.187 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-02-11T09:38:55.270 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-02-11T09:38:55.302 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-02-11T09:38:55.396 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-02-11T09:38:55.397 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-02-11T09:38:55.500 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-02-11T09:38:55.542 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-02-11T09:38:55.627 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-02-11T09:38:55.679 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-02-11T09:38:55.753 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-11T09:38:55.805 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-02-11T09:38:55.878 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-02-11T09:38:55.951 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-02-11T09:38:56.004 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-02-11T09:38:56.098 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-02-11T09:38:56.130 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-02-11T09:38:56.131 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-02-11T09:38:56.213 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-02-11T09:38:56.215 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-02-11T09:38:56.255 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-02-11T09:38:56.257 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-02-11T09:38:56.360 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-02-11T09:38:56.381 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-02-11T09:38:56.496 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-02-11T09:38:56.507 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-02-11T09:38:56.622 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-02-11T09:38:56.644 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-02-11T09:38:56.748 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-02-11T09:38:56.749 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-11T09:38:56.779 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-02-11T09:38:56.884 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-02-11T09:38:56.886 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-02-11T09:38:56.887 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-02-11T09:38:56.916 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-02-11T09:38:57.021 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-02-11T09:38:57.064 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-02-11T09:38:57.136 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-02-11T09:38:57.199 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-11T09:38:57.262 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-02-11T09:38:57.315 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-02-11T09:38:57.388 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-02-11T09:38:57.461 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-02-11T09:38:57.514 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-02-11T09:38:57.587 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-02-11T09:38:57.589 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-02-11T09:38:57.640 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-02-11T09:38:57.642 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-02-11T09:38:57.724 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-02-11T09:38:57.725 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-02-11T09:38:57.766 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-02-11T09:38:57.850 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-02-11T09:38:57.892 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-02-11T09:38:57.987 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-02-11T09:38:58.020 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-02-11T09:38:58.021 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-02-11T09:38:58.134 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-02-11T09:38:58.145 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-02-11T09:38:58.270 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-02-11T09:38:58.270 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-02-11T09:38:58.272 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-11T09:38:58.395 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-02-11T09:38:58.397 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-02-11T09:38:58.407 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-02-11T09:38:58.408 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-02-11T09:38:58.409 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-02-11T09:38:58.511 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-02-11T09:38:58.532 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-02-11T09:38:58.638 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-02-11T09:38:58.671 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-02-11T09:38:58.766 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-11T09:38:58.809 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-02-11T09:38:58.903 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-11T09:38:58.956 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-02-11T09:38:59.030 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-02-11T09:38:59.093 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-02-11T09:38:59.157 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-02-11T09:38:59.241 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-02-11T09:38:59.243 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-02-11T09:38:59.263 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-02-11T09:38:59.357 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-02-11T09:38:59.389 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-02-11T09:38:59.504 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-02-11T09:38:59.515 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-02-11T09:38:59.640 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-02-11T09:38:59.641 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-02-11T09:38:59.643 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-02-11T09:38:59.756 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-02-11T09:38:59.767 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-02-11T09:38:59.881 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-02-11T09:38:59.892 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-02-11T09:39:00.007 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-02-11T09:39:00.028 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-02-11T09:39:00.030 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-02-11T09:39:00.122 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-02-11T09:39:00.155 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-02-11T09:39:00.250 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-02-11T09:39:00.293 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-02-11T09:39:00.377 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-02-11T09:39:00.409 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-11T09:39:00.503 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-02-11T09:39:00.535 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-11T09:39:00.629 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-02-11T09:39:00.671 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-02-11T09:39:00.744 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-02-11T09:39:00.786 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-02-11T09:39:00.870 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-02-11T09:39:00.912 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-02-11T09:39:00.997 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-02-11T09:39:00.998 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-02-11T09:39:01.029 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-02-11T09:39:01.126 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-02-11T09:39:01.158 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-02-11T09:39:01.242 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-02-11T09:39:01.285 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-02-11T09:39:01.369 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-02-11T09:39:01.411 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-02-11T09:39:01.495 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-02-11T09:39:01.547 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-02-11T09:39:01.642 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-02-11T09:39:01.643 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-02-11T09:39:01.685 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-02-11T09:39:01.769 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-02-11T09:39:01.770 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-02-11T09:39:01.801 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-02-11T09:39:01.894 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-02-11T09:39:01.937 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-02-11T09:39:02.021 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-02-11T09:39:38.949 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0O 2021-02-11T09:39:38.950 INFO:teuthology.task.ansible.out:q0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} ok: [smithi02 2021-02-11T09:39:38.952 INFO:teuthology.task.ansible.out:6.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtae 2021-02-11T09:39:38.953 INFO:teuthology.task.ansible.out:CbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh 2021-02-11T09:39:38.954 INFO:teuthology.task.ansible.out:/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAB 2021-02-11T09:39:38.956 INFO:teuthology.task.ansible.out:AQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kA 2021-02-11T09:39:38.957 INFO:teuthology.task.ansible.out:ikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0 2021-02-11T09:39:38.958 INFO:teuthology.task.ansible.out:MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OV 2021-02-11T09:39:38.958 INFO:teuthology.task.ansible.out:j+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-02-11T09:39:38.961 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-02-11T09:39:38.962 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-02-11T09:39:38.962 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/W 2021-02-11T09:39:38.963 INFO:teuthology.task.ansible.out:BFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-02-11T09:39:38.964 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": 2021-02-11T09:39:38.965 INFO:teuthology.task.ansible.out: true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-02-11T09:39:38.966 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiP 2021-02-11T09:39:38.968 INFO:teuthology.task.ansible.out:gIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "cha 2021-02-11T09:39:38.969 INFO:teuthology.task.ansible.out:nged": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => 2021-02-11T09:39:38.970 INFO:teuthology.task.ansible.out:(item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c3 2021-02-11T09:39:38.972 INFO:teuthology.task.ansible.out:2204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": 2021-02-11T09:39:38.973 INFO:teuthology.task.ansible.out:"item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIa 2021-02-11T09:39:38.974 INFO:teuthology.task.ansible.out:O2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4 2021-02-11T09:39:38.976 INFO:teuthology.task.ansible.out:ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'} 2021-02-11T09:39:38.977 INFO:teuthology.task.ansible.out:) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2021-02-11T09:39:38.978 INFO:teuthology.task.ansible.out:": "soumyakoduri", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0 2021-02-11T09:39:38.980 INFO:teuthology.task.ansible.out:BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggE 2021-02-11T09:39:38.981 INFO:teuthology.task.ansible.out:XyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05 2021-02-11T09:39:38.982 INFO:teuthology.task.ansible.out:B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAA 2021-02-11T09:39:38.983 INFO:teuthology.task.ansible.out:ADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539 2021-02-11T09:39:38.985 INFO:teuthology.task.ansible.out:a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShN 2021-02-11T09:39:38.986 INFO:teuthology.task.ansible.out:DsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.local", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8N 2021-02-11T09:39:38.987 INFO:teuthology.task.ansible.out:F3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfb 2021-02-11T09:39:38.989 INFO:teuthology.task.ansible.out:wUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6 2021-02-11T09:39:38.990 INFO:teuthology.task.ansible.out:076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} ok: [smith 2021-02-11T09:39:38.991 INFO:teuthology.task.ansible.out:i026.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJU 2021-02-11T09:39:38.993 INFO:teuthology.task.ansible.out:AQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2ep 2021-02-11T09:39:38.994 INFO:teuthology.task.ansible.out:GeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7 2021-02-11T09:39:38.996 INFO:teuthology.task.ansible.out:elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHy 2021-02-11T09:39:38.997 INFO:teuthology.task.ansible.out:kgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc 2021-02-11T09:39:38.998 INFO:teuthology.task.ansible.out:2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ss 2021-02-11T09:39:39.000 INFO:teuthology.task.ansible.out:h-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vL 2021-02-11T09:39:39.001 INFO:teuthology.task.ansible.out:BkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf5068 2021-02-11T09:39:39.002 INFO:teuthology.task.ansible.out:09f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7 2021-02-11T09:39:39.003 INFO:teuthology.task.ansible.out:f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorize 2021-02-11T09:39:39.005 INFO:teuthology.task.ansible.out:d_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN 2021-02-11T09:39:39.006 INFO:teuthology.task.ansible.out:/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjLd+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABA 2021-02-11T09:39:39.007 INFO:teuthology.task.ansible.out:AABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e51 2021-02-11T09:39:39.009 INFO:teuthology.task.ansible.out:90de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2 2021-02-11T09:39:39.010 INFO:teuthology.task.ansible.out:c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjC 2021-02-11T09:39:39.011 INFO:teuthology.task.ansible.out:dWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_va 2021-02-11T09:39:39.013 INFO:teuthology.task.ansible.out:r": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpa 2021-02-11T09:39:39.014 INFO:teuthology.task.ansible.out:d g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad 2021-02-11T09:39:39.015 INFO:teuthology.task.ansible.out:/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD 2021-02-11T09:39:39.017 INFO:teuthology.task.ansible.out:1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3b 2021-02-11T09:39:39.018 INFO:teuthology.task.ansible.out:bbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48 2021-02-11T09:39:39.019 INFO:teuthology.task.ansible.out:bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7 2021-02-11T09:39:39.021 INFO:teuthology.task.ansible.out:b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjik 2021-02-11T09:39:39.022 INFO:teuthology.task.ansible.out:Nva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VO 2021-02-11T09:39:39.023 INFO:teuthology.task.ansible.out:pTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e 2021-02-11T09:39:39.025 INFO:teuthology.task.ansible.out:13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12c 2021-02-11T09:39:39.026 INFO:teuthology.task.ansible.out:fc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com 2021-02-11T09:39:39.027 INFO:teuthology.task.ansible.out:] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjLd+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVw 2021-02-11T09:39:39.029 INFO:teuthology.task.ansible.out:wmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8r 2021-02-11T09:39:39.030 INFO:teuthology.task.ansible.out:q8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd8 2021-02-11T09:39:39.031 INFO:teuthology.task.ansible.out:5e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={' 2021-02-11T09:39:39.033 INFO:teuthology.task.ansible.out:name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} changed: [smithi026.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl 2021-02-11T09:39:39.034 INFO:teuthology.task.ansible.out:8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-02-11T09:39:39.077 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-02-11T09:39:39.078 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-02-11T09:39:39.079 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-02-11T09:39:39.080 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVey 2021-02-11T09:39:39.081 INFO:teuthology.task.ansible.out:AqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-02-11T09:39:39.081 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ 2021-02-11T09:39:39.082 INFO:teuthology.task.ansible.out:5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-02-11T09:39:39.082 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitn 2021-02-11T09:39:39.084 INFO:teuthology.task.ansible.out:n0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-02-11T09:39:39.084 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key" 2021-02-11T09:39:39.085 INFO:teuthology.task.ansible.out:: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-02-11T09:39:39.086 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3Ad 2021-02-11T09:39:39.088 INFO:teuthology.task.ansible.out:oYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} changed: [smithi133.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e23 2021-02-11T09:39:39.089 INFO:teuthology.task.ansible.out:1a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-02-11T09:39:39.167 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:39 +0000 (0:00:46.924) 0:01:14.732 ***** 2021-02-11T09:39:39.202 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:39 +0000 (0:00:00.035) 0:01:14.767 ***** 2021-02-11T09:39:39.222 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T09:39:39.223 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.234 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.235 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.246 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.247 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.257 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.258 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.259 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.260 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.270 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.271 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.282 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.283 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.293 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.305 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.305 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.431 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-02-11T09:39:39.442 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.442 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-02-11T09:39:39.444 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.454 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.465 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.581 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-02-11T09:39:39.603 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-02-11T09:39:39.697 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-02-11T09:39:39.708 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.719 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.719 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.730 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.741 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.752 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.752 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-02-11T09:39:39.754 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.764 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.765 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.776 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.776 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.777 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.788 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.788 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.799 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.799 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.801 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.811 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.812 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.823 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.823 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.824 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.835 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.836 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.837 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.847 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.847 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.849 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.859 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.859 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.860 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.871 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.871 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.873 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.883 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.884 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.885 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.895 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.895 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.897 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.907 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.907 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.908 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.918 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.919 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.920 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.930 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.930 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.932 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.942 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.942 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.943 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.954 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.954 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.956 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.965 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.966 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.977 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.977 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.978 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.989 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.989 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.990 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.000 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.001 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.002 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.012 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.012 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.014 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.024 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.024 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.025 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.035 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.036 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.037 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.047 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.048 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.059 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.059 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.060 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.071 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.072 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.073 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.083 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.083 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.084 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.094 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.095 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.096 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.106 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.107 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.108 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.118 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.118 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.120 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.130 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.130 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.131 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.142 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.142 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.143 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.154 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.154 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.165 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.165 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.167 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.177 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.177 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.179 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.188 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.189 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.190 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.200 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.200 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.202 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.212 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.212 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.214 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.224 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.225 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.226 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.236 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.236 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.237 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.248 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.248 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.249 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.260 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.260 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.271 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.271 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.273 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.283 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.284 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.285 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.295 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.295 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.297 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.307 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.307 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.309 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.319 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.319 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.330 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.331 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.332 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.342 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.343 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.344 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.354 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.355 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.356 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.366 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.366 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.368 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.378 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.379 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.380 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.390 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.390 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.392 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.402 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.402 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.403 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.413 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.414 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.415 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.425 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.425 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.427 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.437 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.437 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.438 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.449 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.449 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.451 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.460 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.461 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.462 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.472 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.473 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.474 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.484 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.485 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.486 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.496 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.496 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.498 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.507 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.508 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.509 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.519 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.520 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.521 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.531 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.531 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.533 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.543 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.544 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.545 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.555 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.556 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.557 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.567 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.568 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.569 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.579 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.580 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.591 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.591 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.592 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.603 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.603 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.614 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.615 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.616 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.626 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.627 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.638 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.638 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.640 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.650 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.651 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.668 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.669 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.704 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:40 +0000 (0:00:01.502) 0:01:16.269 ***** 2021-02-11T09:39:40.711 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.730 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2021-02-11T09:39:40.731 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.763 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:40 +0000 (0:00:00.058) 0:01:16.328 ***** 2021-02-11T09:39:40.794 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:40 +0000 (0:00:00.031) 0:01:16.359 ***** 2021-02-11T09:39:40.821 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.858 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:40 +0000 (0:00:00.063) 0:01:16.423 ***** 2021-02-11T09:39:40.927 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:39:40.928 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-02-11T09:39:41.022 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-02-11T09:39:41.044 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-02-11T09:39:41.169 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-02-11T09:39:41.170 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-02-11T09:39:41.285 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-02-11T09:39:41.296 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-02-11T09:39:41.411 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-02-11T09:39:41.422 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-02-11T09:39:41.537 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-02-11T09:39:41.559 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-02-11T09:39:41.653 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-02-11T09:39:41.695 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-02-11T09:39:41.779 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-02-11T09:39:41.832 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-02-11T09:39:41.905 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-02-11T09:39:41.948 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-02-11T09:39:42.021 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-02-11T09:39:42.074 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-02-11T09:39:42.148 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-11T09:39:42.179 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-02-11T09:39:42.263 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-02-11T09:39:42.325 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-11T09:39:42.388 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-11T09:39:42.441 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-02-11T09:39:42.515 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-02-11T09:39:42.577 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-11T09:39:42.631 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-11T09:39:42.715 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-02-11T09:39:42.757 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-02-11T09:39:42.841 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-11T09:39:42.883 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-02-11T09:39:42.966 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-02-11T09:39:42.998 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-11T09:39:43.092 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-02-11T09:39:43.124 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-02-11T09:39:43.209 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-11T09:39:43.241 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-02-11T09:39:43.346 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-02-11T09:39:43.368 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-11T09:39:43.482 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-02-11T09:39:43.483 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-02-11T09:39:43.609 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-11T09:39:43.621 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-11T09:39:43.736 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-02-11T09:39:43.747 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-02-11T09:39:43.852 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-02-11T09:39:43.863 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-11T09:39:43.979 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-02-11T09:39:44.001 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-02-11T09:39:44.106 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-02-11T09:39:44.117 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-02-11T09:39:44.234 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-11T09:39:44.257 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-02-11T09:39:44.351 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-02-11T09:39:44.373 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-02-11T09:39:44.470 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-02-11T09:39:44.514 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-11T09:39:44.597 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-02-11T09:39:44.651 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-02-11T09:39:44.725 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-02-11T09:39:44.768 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-02-11T09:39:44.841 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-02-11T09:39:44.904 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-02-11T09:39:44.957 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-02-11T09:39:45.020 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-02-11T09:39:45.083 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-02-11T09:39:45.146 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-02-11T09:39:45.200 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-02-11T09:39:45.273 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-02-11T09:39:45.326 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-02-11T09:39:45.410 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-02-11T09:39:45.452 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-02-11T09:39:45.547 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-02-11T09:39:45.663 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-02-11T09:39:45.779 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-02-11T09:39:45.916 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-02-11T09:39:46.053 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-11T09:39:46.178 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-11T09:39:46.263 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-02-11T09:39:46.307 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-11T09:39:46.382 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-11T09:39:46.425 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-11T09:39:46.519 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-11T09:39:46.551 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-11T09:39:46.645 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-11T09:39:46.667 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-11T09:39:46.771 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-11T09:39:46.793 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-02-11T09:39:46.908 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-11T09:39:46.909 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-11T09:39:47.035 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-11T09:39:47.036 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-02-11T09:39:47.151 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-11T09:39:47.181 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-02-11T09:39:47.228 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:06.370) 0:01:22.793 ***** 2021-02-11T09:39:47.298 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-11T09:39:47.425 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-02-11T09:39:47.562 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-11T09:39:47.615 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.387) 0:01:23.180 ***** 2021-02-11T09:39:47.621 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-02-11T09:39:47.623 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => {"changed": true, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ee06ed012f51c2e3764171d4b9180349", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036387.250496-118485152329920/source", "state": "file", "uid": 0} 2021-02-11T09:39:47.662 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.046) 0:01:23.227 ***** 2021-02-11T09:39:47.690 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-02-11T09:39:47.691 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => { "msg": "Host smithi026.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-02-11T09:39:47.729 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.067) 0:01:23.294 ***** 2021-02-11T09:39:47.767 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-02-11T09:39:47.768 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-02-11T09:39:47.803 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.073) 0:01:23.368 ***** 2021-02-11T09:39:47.829 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-02-11T09:39:47.831 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:47.865 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.061) 0:01:23.430 ***** 2021-02-11T09:39:47.891 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-02-11T09:39:47.892 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:47.928 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.063) 0:01:23.493 ***** 2021-02-11T09:39:47.934 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-02-11T09:39:47.934 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ee06ed012f51c2e3764171d4b9180349", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036387.6371937-101564633258609/source", "state": "file", "uid": 0} 2021-02-11T09:39:47.972 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.043) 0:01:23.537 ***** 2021-02-11T09:39:47.978 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-02-11T09:39:47.979 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:48.017 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.045) 0:01:23.583 ***** 2021-02-11T09:39:48.024 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-02-11T09:39:48.025 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => { "msg": "Host smithi133.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-02-11T09:39:48.062 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.044) 0:01:23.627 ***** 2021-02-11T09:39:48.068 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:48.105 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.043) 0:01:23.670 ***** 2021-02-11T09:39:48.111 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-02-11T09:39:48.112 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-02-11T09:39:48.148 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.043) 0:01:23.714 ***** 2021-02-11T09:39:48.155 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-02-11T09:39:48.156 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:48.191 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.042) 0:01:23.756 ***** 2021-02-11T09:39:48.197 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-02-11T09:39:48.199 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:48.233 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.041) 0:01:23.798 ***** 2021-02-11T09:39:48.239 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-02-11T09:39:48.241 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:48.277 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.043) 0:01:23.842 ***** 2021-02-11T09:39:48.284 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-02-11T09:39:48.286 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:48.320 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.043) 0:01:23.885 ***** 2021-02-11T09:39:48.326 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-02-11T09:39:48.327 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:48.363 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.043) 0:01:23.928 ***** 2021-02-11T09:39:48.370 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-02-11T09:39:48.372 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:48.411 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.047) 0:01:23.976 ***** 2021-02-11T09:39:48.417 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-02-11T09:39:48.419 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:48.453 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.042) 0:01:24.018 ***** 2021-02-11T09:39:48.461 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-02-11T09:39:48.463 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:48.499 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.045) 0:01:24.064 ***** 2021-02-11T09:39:48.505 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-02-11T09:39:48.507 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi026.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:39:48.543 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.043) 0:01:24.108 ***** 2021-02-11T09:39:48.551 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-02-11T09:39:48.552 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:48.586 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.043) 0:01:24.151 ***** 2021-02-11T09:39:48.592 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:48.632 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.046) 0:01:24.197 ***** 2021-02-11T09:39:48.639 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-02-11T09:39:48.641 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:48.675 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.042) 0:01:24.240 ***** 2021-02-11T09:39:48.680 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-02-11T09:39:48.681 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:48.716 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.041) 0:01:24.281 ***** 2021-02-11T09:39:48.723 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-02-11T09:39:48.724 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:48.759 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.043) 0:01:24.325 ***** 2021-02-11T09:39:48.765 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-02-11T09:39:48.765 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:39:48.800 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.040) 0:01:24.365 ***** 2021-02-11T09:39:48.805 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-02-11T09:39:48.806 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:48.839 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.039) 0:01:24.404 ***** 2021-02-11T09:39:48.846 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-02-11T09:39:48.848 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:48.882 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.042) 0:01:24.447 ***** 2021-02-11T09:39:48.888 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-02-11T09:39:48.888 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:39:48.890 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:39:48.924 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.042) 0:01:24.490 ***** 2021-02-11T09:39:48.932 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-02-11T09:39:48.933 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:39:48.967 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.042) 0:01:24.532 ***** 2021-02-11T09:39:48.973 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-02-11T09:39:48.975 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.012 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.045) 0:01:24.577 ***** 2021-02-11T09:39:49.018 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-02-11T09:39:49.020 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.054 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.041) 0:01:24.619 ***** 2021-02-11T09:39:49.061 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-02-11T09:39:49.063 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.096 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.042) 0:01:24.661 ***** 2021-02-11T09:39:49.105 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-02-11T09:39:49.106 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.141 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.044) 0:01:24.706 ***** 2021-02-11T09:39:49.146 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-02-11T09:39:49.147 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:39:49.182 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.041) 0:01:24.747 ***** 2021-02-11T09:39:49.188 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-02-11T09:39:49.188 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.225 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.042) 0:01:24.790 ***** 2021-02-11T09:39:49.231 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-02-11T09:39:49.231 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.267 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.042) 0:01:24.832 ***** 2021-02-11T09:39:49.273 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-02-11T09:39:49.273 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.308 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.041) 0:01:24.874 ***** 2021-02-11T09:39:49.314 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-02-11T09:39:49.314 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:39:49.350 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.041) 0:01:24.915 ***** 2021-02-11T09:39:49.356 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-02-11T09:39:49.357 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:39:49.392 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.042) 0:01:24.957 ***** 2021-02-11T09:39:49.401 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-02-11T09:39:49.402 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.437 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.044) 0:01:25.002 ***** 2021-02-11T09:39:49.443 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-11T09:39:49.444 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.480 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.043) 0:01:25.045 ***** 2021-02-11T09:39:49.487 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-02-11T09:39:49.487 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.523 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.042) 0:01:25.088 ***** 2021-02-11T09:39:49.529 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-02-11T09:39:49.530 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.566 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.042) 0:01:25.131 ***** 2021-02-11T09:39:49.572 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-02-11T09:39:49.572 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.611 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.045) 0:01:25.176 ***** 2021-02-11T09:39:49.617 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-02-11T09:39:49.617 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.654 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.042) 0:01:25.219 ***** 2021-02-11T09:39:49.661 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-02-11T09:39:49.662 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.698 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.043) 0:01:25.263 ***** 2021-02-11T09:39:49.704 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-02-11T09:39:49.705 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.740 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.041) 0:01:25.305 ***** 2021-02-11T09:39:49.746 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-02-11T09:39:49.746 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:39:49.786 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.045) 0:01:25.351 ***** 2021-02-11T09:39:49.792 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-02-11T09:39:49.793 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.841 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.055) 0:01:25.407 ***** 2021-02-11T09:39:49.852 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-11T09:39:49.852 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.888 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.046) 0:01:25.453 ***** 2021-02-11T09:39:49.893 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-02-11T09:39:49.895 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.945 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.057) 0:01:25.510 ***** 2021-02-11T09:39:49.952 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-02-11T09:39:49.953 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:49.989 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:49 +0000 (0:00:00.043) 0:01:25.554 ***** 2021-02-11T09:39:49.996 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-02-11T09:39:49.996 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:39:50.032 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.043) 0:01:25.597 ***** 2021-02-11T09:39:50.038 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-02-11T09:39:50.039 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.075 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.043) 0:01:25.640 ***** 2021-02-11T09:39:50.082 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-02-11T09:39:50.083 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.124 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.048) 0:01:25.689 ***** 2021-02-11T09:39:50.130 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-02-11T09:39:50.132 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.168 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.043) 0:01:25.733 ***** 2021-02-11T09:39:50.174 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-02-11T09:39:50.175 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.210 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.042) 0:01:25.775 ***** 2021-02-11T09:39:50.216 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-02-11T09:39:50.217 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.254 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.044) 0:01:25.819 ***** 2021-02-11T09:39:50.261 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-02-11T09:39:50.263 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.298 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.043) 0:01:25.863 ***** 2021-02-11T09:39:50.306 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-02-11T09:39:50.307 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.344 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.046) 0:01:25.909 ***** 2021-02-11T09:39:50.351 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-02-11T09:39:50.351 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.353 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.389 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.045) 0:01:25.955 ***** 2021-02-11T09:39:50.397 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-02-11T09:39:50.399 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:39:50.434 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.044) 0:01:25.999 ***** 2021-02-11T09:39:50.440 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-11T09:39:50.441 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.442 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.477 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.043) 0:01:26.043 ***** 2021-02-11T09:39:50.485 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-02-11T09:39:50.486 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.520 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.042) 0:01:26.085 ***** 2021-02-11T09:39:50.525 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-11T09:39:50.527 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.561 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.041) 0:01:26.126 ***** 2021-02-11T09:39:50.567 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-02-11T09:39:50.569 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.606 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.044) 0:01:26.171 ***** 2021-02-11T09:39:50.633 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-02-11T09:39:50.635 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.668 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.061) 0:01:26.233 ***** 2021-02-11T09:39:50.705 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-02-11T09:39:50.707 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} TASK [common : Get the current timezone] *************************************** ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.016654", "end": "2021-02-11 09:39:50.701335", "rc": 0, "start": "2021-02-11 09:39:50.684681", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-02-11T09:39:50.708 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.743 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.074) 0:01:26.308 ***** 2021-02-11T09:39:50.786 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.043) 0:01:26.351 ***** 2021-02-11T09:39:50.792 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-02-11T09:39:50.793 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.829 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.042) 0:01:26.394 ***** 2021-02-11T09:39:50.834 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-11T09:39:50.834 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.836 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.875 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.045) 0:01:26.440 ***** 2021-02-11T09:39:50.881 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-02-11T09:39:50.882 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:50.917 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:50 +0000 (0:00:00.042) 0:01:26.482 ***** 2021-02-11T09:39:51.049 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-11T09:39:51.051 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:51.085 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:51 +0000 (0:00:00.167) 0:01:26.650 ***** 2021-02-11T09:39:51.091 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-02-11T09:39:51.092 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi026.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:51.103 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:51.103 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:51.140 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:51 +0000 (0:00:00.055) 0:01:26.705 ***** 2021-02-11T09:39:51.168 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-11T09:39:51.170 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:51.203 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:51 +0000 (0:00:00.062) 0:01:26.768 ***** 2021-02-11T09:39:51.231 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-02-11T09:39:51.232 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Get the current timezone] *************************************** ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.012610", "end": "2021-02-11 09:39:51.545994", "rc": 0, "start": "2021-02-11 09:39:51.533384", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-02-11T09:39:51.266 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:51 +0000 (0:00:00.063) 0:01:26.831 ***** 2021-02-11T09:39:51.300 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:51 +0000 (0:00:00.033) 0:01:26.865 ***** 2021-02-11T09:39:51.306 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-02-11T09:39:51.308 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:51.342 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:51 +0000 (0:00:00.042) 0:01:26.907 ***** 2021-02-11T09:39:51.349 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-02-11T09:39:51.350 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:51.386 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:51 +0000 (0:00:00.043) 0:01:26.951 ***** 2021-02-11T09:39:51.391 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-02-11T09:39:51.393 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:51.427 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:51 +0000 (0:00:00.041) 0:01:26.992 ***** 2021-02-11T09:39:51.433 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-02-11T09:39:51.434 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:51.468 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:51 +0000 (0:00:00.040) 0:01:27.033 ***** 2021-02-11T09:39:51.474 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-02-11T09:39:51.476 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:51.526 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:51 +0000 (0:00:00.058) 0:01:27.092 ***** 2021-02-11T09:39:51.533 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-02-11T09:39:51.533 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:51.535 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:51.536 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:51.574 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:51 +0000 (0:00:00.047) 0:01:27.139 ***** 2021-02-11T09:39:51.601 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-11T09:39:51.602 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:51.637 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:51 +0000 (0:00:00.062) 0:01:27.202 ***** 2021-02-11T09:39:51.664 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-02-11T09:39:51.665 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:51.703 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:51 +0000 (0:00:00.066) 0:01:27.268 ***** 2021-02-11T09:39:51.729 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-02-11T09:39:51.731 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:51.762 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:51 +0000 (0:00:00.059) 0:01:27.328 ***** 2021-02-11T09:39:51.788 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:51.823 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:51 +0000 (0:00:00.060) 0:01:27.388 ***** 2021-02-11T09:39:51.849 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-02-11T09:39:51.851 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:51.883 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:51 +0000 (0:00:00.060) 0:01:27.449 ***** 2021-02-11T09:39:52.190 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-02-11T09:39:52.192 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.184390", "end": "2021-02-11 09:39:52.500389", "rc": 0, "start": "2021-02-11 09:39:52.315999", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:52.229 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:52 +0000 (0:00:00.345) 0:01:27.794 ***** 2021-02-11T09:39:52.255 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-02-11T09:39:52.256 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:52.291 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:52 +0000 (0:00:00.062) 0:01:27.857 ***** 2021-02-11T09:39:52.475 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-02-11T09:39:52.477 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.049106", "end": "2021-02-11 09:39:52.781127", "rc": 0, "start": "2021-02-11 09:39:52.732021", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:52.511 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:52 +0000 (0:00:00.219) 0:01:28.077 ***** 2021-02-11T09:39:52.517 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-02-11T09:39:52.517 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.860002", "end": "2021-02-11 09:39:52.500375", "rc": 0, "start": "2021-02-11 09:39:51.640373", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:52.556 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:52 +0000 (0:00:00.044) 0:01:28.121 ***** 2021-02-11T09:39:52.582 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-02-11T09:39:52.583 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:52.615 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:52 +0000 (0:00:00.059) 0:01:28.180 ***** 2021-02-11T09:39:52.807 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-02-11T09:39:52.807 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.056863", "end": "2021-02-11 09:39:52.788970", "rc": 0, "start": "2021-02-11 09:39:52.732107", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:52.843 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:52 +0000 (0:00:00.227) 0:01:28.408 ***** 2021-02-11T09:39:56.104 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-02-11T09:39:56.106 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1613036134, "cache_updated": false, "changed": false} 2021-02-11T09:39:56.140 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:56 +0000 (0:00:03.297) 0:01:31.705 ***** 2021-02-11T09:39:56.624 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1613036091, "cache_updated": false, "changed": false} 2021-02-11T09:39:56.660 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:56 +0000 (0:00:00.520) 0:01:32.225 ***** 2021-02-11T09:39:57.007 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-02-11T09:39:57.009 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1613036134, "cache_updated": false, "changed": false} 2021-02-11T09:39:57.044 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:57 +0000 (0:00:00.384) 0:01:32.609 ***** 2021-02-11T09:39:57.071 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-02-11T09:39:57.072 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:57.106 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:57 +0000 (0:00:00.061) 0:01:32.671 ***** 2021-02-11T09:39:57.525 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-02-11T09:39:57.526 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-02-11T09:39:57.561 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:57 +0000 (0:00:00.455) 0:01:33.126 ***** 2021-02-11T09:39:57.672 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-02-11T09:39:57.674 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"cache_update_time": 1613036091, "cache_updated": false, "changed": false} 2021-02-11T09:39:57.730 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:57 +0000 (0:00:00.169) 0:01:33.295 ***** 2021-02-11T09:39:57.760 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-02-11T09:39:57.762 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:57.819 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:57 +0000 (0:00:00.088) 0:01:33.384 ***** 2021-02-11T09:39:58.143 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-02-11T09:39:58.144 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-02-11T09:39:58.179 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:58 +0000 (0:00:00.359) 0:01:33.744 ***** 2021-02-11T09:39:58.450 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-02-11T09:39:58.452 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1613036134, "cache_updated": false, "changed": false} 2021-02-11T09:39:58.487 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:58 +0000 (0:00:00.308) 0:01:34.053 ***** 2021-02-11T09:39:58.828 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-11T09:39:58.829 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-02-11T09:39:59.068 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-02-11T09:39:59.070 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"cache_update_time": 1613036091, "cache_updated": false, "changed": false} 2021-02-11T09:39:59.109 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:59 +0000 (0:00:00.621) 0:01:34.674 ***** 2021-02-11T09:39:59.138 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-11T09:39:59.139 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-02-11T09:39:59.419 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-02-11T09:39:59.457 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:59 +0000 (0:00:00.348) 0:01:35.022 ***** 2021-02-11T09:39:59.463 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-02-11T09:39:59.579 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-02-11T09:39:59.580 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:39:59.616 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:59 +0000 (0:00:00.158) 0:01:35.181 ***** 2021-02-11T09:39:59.735 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-11T09:39:59.736 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-02-11T09:39:59.913 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-11T09:39:59.915 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-02-11T09:39:59.987 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-11T09:39:59.988 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-02-11T09:40:00.025 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:00 +0000 (0:00:00.409) 0:01:35.590 ***** 2021-02-11T09:40:00.134 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-02-11T09:40:00.136 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:40:00.171 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:00 +0000 (0:00:00.146) 0:01:35.736 ***** 2021-02-11T09:40:00.178 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-11T09:40:00.178 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-02-11T09:40:00.430 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-02-11T09:40:00.467 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:00 +0000 (0:00:00.295) 0:01:36.032 ***** 2021-02-11T09:40:00.473 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-02-11T09:40:00.496 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-02-11T09:40:00.497 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-02-11T09:40:00.529 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:00 +0000 (0:00:00.062) 0:01:36.095 ***** 2021-02-11T09:40:00.555 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-02-11T09:40:00.557 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:00.589 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:00 +0000 (0:00:00.058) 0:01:36.154 ***** 2021-02-11T09:40:00.613 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-02-11T09:40:00.615 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:40:00.647 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:00 +0000 (0:00:00.059) 0:01:36.213 ***** 2021-02-11T09:40:00.683 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-02-11T09:40:00.684 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:00.694 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:00.705 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:00.705 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:00.740 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:00 +0000 (0:00:00.092) 0:01:36.305 ***** 2021-02-11T09:40:00.745 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-11T09:40:00.745 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-02-11T09:40:00.767 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-02-11T09:40:00.769 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:00.804 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:00 +0000 (0:00:00.064) 0:01:36.370 ***** 2021-02-11T09:40:00.965 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-11T09:40:00.966 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-02-11T09:40:01.002 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:01 +0000 (0:00:00.197) 0:01:36.567 ***** 2021-02-11T09:40:01.039 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-02-11T09:40:01.041 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-02-11T09:40:01.072 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:01 +0000 (0:00:00.069) 0:01:36.637 ***** 2021-02-11T09:40:01.100 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:01.132 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:01 +0000 (0:00:00.060) 0:01:36.697 ***** 2021-02-11T09:40:01.159 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-02-11T09:40:01.160 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:40:01.192 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:01 +0000 (0:00:00.059) 0:01:36.757 ***** 2021-02-11T09:40:01.228 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-02-11T09:40:01.229 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:01.240 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:01.240 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:01.251 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:01.285 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:01 +0000 (0:00:00.092) 0:01:36.850 ***** 2021-02-11T09:40:01.311 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-02-11T09:40:01.312 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:01.345 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:01 +0000 (0:00:00.059) 0:01:36.910 ***** 2021-02-11T09:40:01.756 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-02-11T09:40:01.758 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1613036134, "cache_updated": false, "changed": false} 2021-02-11T09:40:01.791 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:01 +0000 (0:00:00.446) 0:01:37.356 ***** 2021-02-11T09:40:02.096 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-02-11T09:40:02.098 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-02-11T09:40:02.129 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:02 +0000 (0:00:00.338) 0:01:37.695 ***** 2021-02-11T09:40:02.280 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-02-11T09:40:02.281 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"cache_update_time": 1613036091, "cache_updated": false, "changed": false} 2021-02-11T09:40:02.315 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:02 +0000 (0:00:00.185) 0:01:37.880 ***** 2021-02-11T09:40:02.425 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-02-11T09:40:02.426 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-02-11T09:40:02.490 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:02 +0000 (0:00:00.174) 0:01:38.055 ***** 2021-02-11T09:40:02.612 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-02-11T09:40:02.613 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-02-11T09:40:02.651 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:02 +0000 (0:00:00.161) 0:01:38.216 ***** 2021-02-11T09:40:02.657 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-02-11T09:40:02.657 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-02-11T09:40:02.741 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-02-11T09:40:02.779 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:02 +0000 (0:00:00.128) 0:01:38.344 ***** 2021-02-11T09:40:02.817 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-02-11T09:40:02.818 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:02.853 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:02 +0000 (0:00:00.073) 0:01:38.418 ***** 2021-02-11T09:40:02.880 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-02-11T09:40:02.881 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:40:02.915 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:02 +0000 (0:00:00.062) 0:01:38.480 ***** 2021-02-11T09:40:02.931 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-02-11T09:40:02.933 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-02-11T09:40:02.965 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:02 +0000 (0:00:00.049) 0:01:38.530 ***** 2021-02-11T09:40:02.970 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-02-11T09:40:02.971 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:03.006 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:03 +0000 (0:00:00.041) 0:01:38.571 ***** 2021-02-11T09:40:03.094 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-02-11T09:40:03.096 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-02-11T09:40:03.199 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-02-11T09:40:03.235 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:03 +0000 (0:00:00.229) 0:01:38.800 ***** 2021-02-11T09:40:03.261 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-02-11T09:40:03.263 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:03.294 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:03 +0000 (0:00:00.059) 0:01:38.859 ***** 2021-02-11T09:40:03.320 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-02-11T09:40:03.322 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-02-11T09:40:03.355 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:03 +0000 (0:00:00.060) 0:01:38.920 ***** 2021-02-11T09:40:03.388 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:03 +0000 (0:00:00.032) 0:01:38.953 ***** 2021-02-11T09:40:03.393 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-02-11T09:40:03.394 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:03.429 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:03 +0000 (0:00:00.040) 0:01:38.994 ***** 2021-02-11T09:40:03.740 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-02-11T09:40:03.741 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-02-11T09:40:03.774 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:03 +0000 (0:00:00.345) 0:01:39.339 ***** 2021-02-11T09:40:03.987 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-02-11T09:40:03.989 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 09:37:51 UTC", "ActiveEnterTimestampMonotonic": "18719109", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket local-fs.target var-run.mount systemd-tmpfiles-setup.service basic.target system.slice remote-fs.target network.target sysinit.target -.mount nss-lookup.target time-sync.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 09:37:51 UTC", "AssertTimestampMonotonic": "18718355", "Before": "xdm.service getty@tty1.service multi-user.target shutdown.target plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 09:37:51 UTC", "ConditionTimestampMonotonic": "18718353", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-02-11T09:40:03.990 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1242", "ExecMainStartTimestamp": "Thu 2021-02-11 09:37:51 UTC", "ExecMainStartTimestampMonotonic": "18719076", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2021-02-11 09:37:51 UTC] ; stop_time=[n/a] ; pid=1242 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 09:37:51 UTC", "InactiveExitTimestampMonotonic": "18719109", "InvocationID": "b79e90e089c04cb5ac3759e7f6a4372c", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2021-02-11T09:40:03.991 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1242", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2021-02-11T09:40:03.993 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 09:37:51 UTC", "StateChangeTimestampMonotonic": "18719109", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 09:37:51 UTC", "WatchdogTimestampMonotonic": "18719108", "WatchdogUSec": "0"}} 2021-02-11T09:40:04.025 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.251) 0:01:39.591 ***** 2021-02-11T09:40:04.033 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 09:38:12 UTC", "ActiveEnterTimestampMonotonic": "21644480", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target -.mount basic.target systemd-journald.socket system.slice nss-lookup.target remote-fs.target local-fs.target network.target time-sync.target systemd-tmpfiles-setup.service var-run.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 09:38:12 UTC", "AssertTimestampMonotonic": "21643424", "Before": "getty@tty1.service shutdown.target plymouth-quit.service multi-user.target xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 09:38:12 UTC", "ConditionTimestampMonotonic": "21643424", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-02-11T09:40:04.034 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1288", "ExecMainStartTimestamp": "Thu 2021-02-11 09:38:12 UTC", "ExecMainStartTimestampMonotonic": "21644447", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2021-02-11 09:38:12 UTC] ; stop_time=[n/a] ; pid=1288 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 09:38:12 UTC", "InactiveExitTimestampMonotonic": "21644480", "InvocationID": "327d68eac8fc4640bdc3a29209bf9d9c", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2021-02-11T09:40:04.036 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1288", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2021-02-11T09:40:04.037 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 09:38:12 UTC", "StateChangeTimestampMonotonic": "21644480", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 09:38:12 UTC", "WatchdogTimestampMonotonic": "21644480", "WatchdogUSec": "0"}} 2021-02-11T09:40:04.068 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.043) 0:01:39.634 ***** 2021-02-11T09:40:04.074 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-02-11T09:40:04.075 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.109 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.040) 0:01:39.674 ***** 2021-02-11T09:40:04.115 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.150 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.040) 0:01:39.715 ***** 2021-02-11T09:40:04.156 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-02-11T09:40:04.157 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.192 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.042) 0:01:39.757 ***** 2021-02-11T09:40:04.199 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.235 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.043) 0:01:39.800 ***** 2021-02-11T09:40:04.240 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-02-11T09:40:04.241 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.276 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.040) 0:01:39.841 ***** 2021-02-11T09:40:04.282 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.318 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.041) 0:01:39.883 ***** 2021-02-11T09:40:04.323 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-02-11T09:40:04.324 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.359 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.041) 0:01:39.924 ***** 2021-02-11T09:40:04.365 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.402 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.042) 0:01:39.967 ***** 2021-02-11T09:40:04.408 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-02-11T09:40:04.408 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.445 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.043) 0:01:40.010 ***** 2021-02-11T09:40:04.460 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.496 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.050) 0:01:40.061 ***** 2021-02-11T09:40:04.502 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-02-11T09:40:04.502 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.539 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.043) 0:01:40.104 ***** 2021-02-11T09:40:04.546 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.582 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.043) 0:01:40.147 ***** 2021-02-11T09:40:04.588 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-02-11T09:40:04.589 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.624 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.042) 0:01:40.190 ***** 2021-02-11T09:40:04.631 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.668 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.043) 0:01:40.233 ***** 2021-02-11T09:40:04.674 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-02-11T09:40:04.675 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.711 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.042) 0:01:40.276 ***** 2021-02-11T09:40:04.717 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.753 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.042) 0:01:40.318 ***** 2021-02-11T09:40:04.759 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-02-11T09:40:04.761 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.794 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.041) 0:01:40.360 ***** 2021-02-11T09:40:04.800 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.836 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.041) 0:01:40.401 ***** 2021-02-11T09:40:04.841 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-02-11T09:40:04.843 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.877 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.041) 0:01:40.442 ***** 2021-02-11T09:40:04.883 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:04.920 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.043) 0:01:40.485 ***** 2021-02-11T09:40:04.926 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-02-11T09:40:04.927 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-02-11T09:40:04.962 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:04 +0000 (0:00:00.042) 0:01:40.528 ***** 2021-02-11T09:40:04.969 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-02-11T09:40:05.005 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:05 +0000 (0:00:00.042) 0:01:40.571 ***** 2021-02-11T09:40:05.011 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-02-11T09:40:05.011 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-02-11T09:40:05.052 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:05 +0000 (0:00:00.046) 0:01:40.617 ***** 2021-02-11T09:40:05.059 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-02-11T09:40:05.097 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:05 +0000 (0:00:00.044) 0:01:40.662 ***** 2021-02-11T09:40:05.102 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-02-11T09:40:05.102 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-02-11T09:40:05.140 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:05 +0000 (0:00:00.042) 0:01:40.705 ***** 2021-02-11T09:40:05.146 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-02-11T09:40:05.181 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:05 +0000 (0:00:00.041) 0:01:40.747 ***** 2021-02-11T09:40:05.186 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-02-11T09:40:05.187 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-11T09:40:05.222 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:05 +0000 (0:00:00.040) 0:01:40.787 ***** 2021-02-11T09:40:05.228 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-11T09:40:05.267 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:05 +0000 (0:00:00.045) 0:01:40.833 ***** 2021-02-11T09:40:05.346 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-02-11T09:40:05.347 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-02-11T09:40:05.382 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:05 +0000 (0:00:00.114) 0:01:40.947 ***** 2021-02-11T09:40:05.398 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-02-11T09:40:05.434 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:05 +0000 (0:00:00.052) 0:01:40.999 ***** 2021-02-11T09:40:05.514 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-02-11T09:40:05.516 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-02-11T09:40:05.551 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:05 +0000 (0:00:00.116) 0:01:41.116 ***** 2021-02-11T09:40:05.568 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-02-11T09:40:05.601 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:05 +0000 (0:00:00.049) 0:01:41.166 ***** 2021-02-11T09:40:05.606 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-02-11T09:40:05.608 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:05.643 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:05 +0000 (0:00:00.041) 0:01:41.208 ***** 2021-02-11T09:40:05.650 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:05.684 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:05 +0000 (0:00:00.041) 0:01:41.249 ***** 2021-02-11T09:40:05.690 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-02-11T09:40:05.691 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:05.725 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:05 +0000 (0:00:00.040) 0:01:41.290 ***** 2021-02-11T09:40:05.730 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:05.771 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:05 +0000 (0:00:00.045) 0:01:41.336 ***** 2021-02-11T09:40:05.881 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-02-11T09:40:05.882 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-02-11T09:40:05.917 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:05 +0000 (0:00:00.146) 0:01:41.482 ***** 2021-02-11T09:40:05.924 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-02-11T09:40:05.959 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:05 +0000 (0:00:00.041) 0:01:41.524 ***** 2021-02-11T09:40:06.285 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-02-11T09:40:06.286 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-02-11T09:40:06.322 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:06 +0000 (0:00:00.363) 0:01:41.888 ***** 2021-02-11T09:40:06.329 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-02-11T09:40:06.365 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:06 +0000 (0:00:00.042) 0:01:41.930 ***** 2021-02-11T09:40:06.371 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-02-11T09:40:06.371 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi026"}, "changed": false} 2021-02-11T09:40:06.407 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:06 +0000 (0:00:00.041) 0:01:41.972 ***** 2021-02-11T09:40:06.414 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi133"}, "changed": false} 2021-02-11T09:40:06.450 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:06 +0000 (0:00:00.042) 0:01:42.015 ***** 2021-02-11T09:40:07.223 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-02-11T09:40:07.224 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi026.front.sepia.ceph.com", "ansible_hostname": "smithi026", "ansible_nodename": "smithi026"}, "changed": true, "name": "smithi026"} 2021-02-11T09:40:07.257 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:07 +0000 (0:00:00.807) 0:01:42.823 ***** 2021-02-11T09:40:07.264 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi133.front.sepia.ceph.com", "ansible_hostname": "smithi133", "ansible_nodename": "smithi133"}, "changed": true, "name": "smithi133"} 2021-02-11T09:40:07.300 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:07 +0000 (0:00:00.042) 0:01:42.865 ***** 2021-02-11T09:40:07.555 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-02-11T09:40:07.556 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-02-11T09:40:07.594 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:07 +0000 (0:00:00.294) 0:01:43.160 ***** 2021-02-11T09:40:07.622 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-02-11T09:40:07.664 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:07 +0000 (0:00:00.068) 0:01:43.229 ***** 2021-02-11T09:40:07.897 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-02-11T09:40:07.898 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-02-11T09:40:07.934 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:07 +0000 (0:00:00.270) 0:01:43.499 ***** 2021-02-11T09:40:07.982 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-02-11T09:40:08.018 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:08 +0000 (0:00:00.084) 0:01:43.583 ***** 2021-02-11T09:40:08.471 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-02-11T09:40:08.473 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-02-11T09:40:08.507 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:08 +0000 (0:00:00.488) 0:01:44.072 ***** 2021-02-11T09:40:08.535 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:08.569 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:08 +0000 (0:00:00.061) 0:01:44.134 ***** 2021-02-11T09:40:08.575 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-02-11T09:40:08.576 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-02-11T09:40:08.610 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:08 +0000 (0:00:00.041) 0:01:44.175 ***** 2021-02-11T09:40:08.617 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-11T09:40:08.618 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:08.651 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:08 +0000 (0:00:00.041) 0:01:44.217 ***** 2021-02-11T09:40:08.657 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-02-11T09:40:08.659 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:08.695 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:08 +0000 (0:00:00.043) 0:01:44.260 ***** 2021-02-11T09:40:08.701 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-11T09:40:08.702 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:08.738 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:08 +0000 (0:00:00.042) 0:01:44.303 ***** 2021-02-11T09:40:08.744 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-11T09:40:08.745 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:08.779 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:08 +0000 (0:00:00.041) 0:01:44.344 ***** 2021-02-11T09:40:08.785 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-11T09:40:08.787 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:08.826 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:08 +0000 (0:00:00.046) 0:01:44.391 ***** 2021-02-11T09:40:08.832 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-11T09:40:08.832 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:08.869 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:08 +0000 (0:00:00.043) 0:01:44.434 ***** 2021-02-11T09:40:08.876 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-11T09:40:08.877 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:08.911 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:08 +0000 (0:00:00.041) 0:01:44.476 ***** 2021-02-11T09:40:08.917 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-11T09:40:08.918 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:08.953 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:08 +0000 (0:00:00.042) 0:01:44.518 ***** 2021-02-11T09:40:08.959 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-11T09:40:08.959 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:08.993 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:08 +0000 (0:00:00.040) 0:01:44.558 ***** 2021-02-11T09:40:08.999 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-11T09:40:09.000 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:09.035 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.041) 0:01:44.600 ***** 2021-02-11T09:40:09.040 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-11T09:40:09.042 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:09.076 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.041) 0:01:44.642 ***** 2021-02-11T09:40:09.082 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-11T09:40:09.083 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:09.118 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.041) 0:01:44.684 ***** 2021-02-11T09:40:09.124 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-02-11T09:40:09.125 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:09.159 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.040) 0:01:44.724 ***** 2021-02-11T09:40:09.166 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-11T09:40:09.168 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:09.204 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.044) 0:01:44.769 ***** 2021-02-11T09:40:09.210 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-02-11T09:40:09.246 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.042) 0:01:44.811 ***** 2021-02-11T09:40:09.252 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-02-11T09:40:09.253 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:09.288 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.041) 0:01:44.853 ***** 2021-02-11T09:40:09.293 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-02-11T09:40:09.328 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.040) 0:01:44.893 ***** 2021-02-11T09:40:09.334 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-02-11T09:40:09.369 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.040) 0:01:44.934 ***** 2021-02-11T09:40:09.374 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-02-11T09:40:09.412 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.043) 0:01:44.977 ***** 2021-02-11T09:40:09.418 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-02-11T09:40:09.453 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.040) 0:01:45.018 ***** 2021-02-11T09:40:09.461 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-02-11T09:40:09.462 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:09.496 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.042) 0:01:45.061 ***** 2021-02-11T09:40:09.502 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-02-11T09:40:09.542 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.045) 0:01:45.107 ***** 2021-02-11T09:40:09.548 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-02-11T09:40:09.548 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:09.549 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:09.585 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.043) 0:01:45.151 ***** 2021-02-11T09:40:09.592 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-02-11T09:40:09.593 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:09.629 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.043) 0:01:45.194 ***** 2021-02-11T09:40:09.635 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-02-11T09:40:09.636 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:09.673 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.043) 0:01:45.238 ***** 2021-02-11T09:40:09.678 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-02-11T09:40:09.680 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:09.716 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.043) 0:01:45.281 ***** 2021-02-11T09:40:09.722 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:40:09.723 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:09.758 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.042) 0:01:45.323 ***** 2021-02-11T09:40:09.765 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-02-11T09:40:09.766 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:09.798 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.040) 0:01:45.363 ***** 2021-02-11T09:40:09.803 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-02-11T09:40:09.804 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:09.838 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.039) 0:01:45.403 ***** 2021-02-11T09:40:09.843 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:40:09.844 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:09.879 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.041) 0:01:45.445 ***** 2021-02-11T09:40:09.886 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-02-11T09:40:09.886 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:09.921 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.041) 0:01:45.486 ***** 2021-02-11T09:40:09.926 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-02-11T09:40:09.926 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:09.960 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:09 +0000 (0:00:00.038) 0:01:45.525 ***** 2021-02-11T09:40:09.965 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-02-11T09:40:09.965 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.004 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.044) 0:01:45.569 ***** 2021-02-11T09:40:10.010 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-02-11T09:40:10.012 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.044 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.039) 0:01:45.609 ***** 2021-02-11T09:40:10.049 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-02-11T09:40:10.051 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.083 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.039) 0:01:45.648 ***** 2021-02-11T09:40:10.089 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-02-11T09:40:10.090 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.125 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.041) 0:01:45.690 ***** 2021-02-11T09:40:10.130 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-11T09:40:10.131 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.164 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.039) 0:01:45.730 ***** 2021-02-11T09:40:10.169 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-02-11T09:40:10.171 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.204 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.040) 0:01:45.770 ***** 2021-02-11T09:40:10.210 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-11T09:40:10.212 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.246 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.041) 0:01:45.812 ***** 2021-02-11T09:40:10.252 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-11T09:40:10.253 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.287 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.040) 0:01:45.852 ***** 2021-02-11T09:40:10.292 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T09:40:10.293 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.327 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.040) 0:01:45.892 ***** 2021-02-11T09:40:10.334 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-11T09:40:10.335 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.368 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.041) 0:01:45.934 ***** 2021-02-11T09:40:10.374 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-02-11T09:40:10.376 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.409 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.040) 0:01:45.974 ***** 2021-02-11T09:40:10.414 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T09:40:10.416 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.449 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.039) 0:01:46.014 ***** 2021-02-11T09:40:10.454 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-11T09:40:10.456 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.492 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.043) 0:01:46.057 ***** 2021-02-11T09:40:10.499 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-02-11T09:40:10.499 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.535 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.042) 0:01:46.100 ***** 2021-02-11T09:40:10.540 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T09:40:10.541 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.576 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.041) 0:01:46.141 ***** 2021-02-11T09:40:10.581 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-11T09:40:10.582 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.616 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.039) 0:01:46.181 ***** 2021-02-11T09:40:10.621 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-02-11T09:40:10.623 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.655 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.039) 0:01:46.220 ***** 2021-02-11T09:40:10.660 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T09:40:10.662 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.696 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.040) 0:01:46.261 ***** 2021-02-11T09:40:10.700 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-02-11T09:40:10.702 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.736 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.040) 0:01:46.301 ***** 2021-02-11T09:40:10.741 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-02-11T09:40:10.742 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.777 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.040) 0:01:46.342 ***** 2021-02-11T09:40:10.781 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-02-11T09:40:10.783 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.815 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.038) 0:01:46.380 ***** 2021-02-11T09:40:10.820 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-02-11T09:40:10.821 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.855 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.040) 0:01:46.420 ***** 2021-02-11T09:40:10.860 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-02-11T09:40:10.862 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.896 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.040) 0:01:46.461 ***** 2021-02-11T09:40:10.903 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-02-11T09:40:10.904 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.937 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.040) 0:01:46.502 ***** 2021-02-11T09:40:10.942 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-02-11T09:40:10.943 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:10.977 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:10 +0000 (0:00:00.040) 0:01:46.542 ***** 2021-02-11T09:40:10.982 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-02-11T09:40:10.984 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:11.017 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:11 +0000 (0:00:00.040) 0:01:46.582 ***** 2021-02-11T09:40:11.022 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-02-11T09:40:11.024 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:11.057 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:11 +0000 (0:00:00.039) 0:01:46.622 ***** 2021-02-11T09:40:11.063 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-02-11T09:40:11.064 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:11.098 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:11 +0000 (0:00:00.040) 0:01:46.663 ***** 2021-02-11T09:40:11.103 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-02-11T09:40:11.103 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:11.142 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:11 +0000 (0:00:00.044) 0:01:46.707 ***** 2021-02-11T09:40:11.148 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-02-11T09:40:11.149 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:11.184 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:11 +0000 (0:00:00.041) 0:01:46.749 ***** 2021-02-11T09:40:11.191 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-02-11T09:40:11.192 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:11.224 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:11 +0000 (0:00:00.039) 0:01:46.789 ***** 2021-02-11T09:40:11.229 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-02-11T09:40:11.230 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:11.266 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:11 +0000 (0:00:00.041) 0:01:46.831 ***** 2021-02-11T09:40:11.294 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-02-11T09:40:11.295 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:11.327 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:11 +0000 (0:00:00.061) 0:01:46.892 ***** 2021-02-11T09:40:11.353 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-02-11T09:40:11.355 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005138", "end": "2021-02-11 09:40:11.351110", "rc": 0, "start": "2021-02-11 09:40:11.345972", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:40:11.388 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:11 +0000 (0:00:00.061) 0:01:46.954 ***** 2021-02-11T09:40:11.415 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-02-11T09:40:11.448 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:11 +0000 (0:00:00.059) 0:01:47.013 ***** 2021-02-11T09:40:11.474 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-02-11T09:40:11.475 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.004746", "end": "2021-02-11 09:40:11.775155", "rc": 0, "start": "2021-02-11 09:40:11.770409", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:40:11.510 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:11 +0000 (0:00:00.062) 0:01:47.075 ***** 2021-02-11T09:40:11.537 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-02-11T09:40:11.570 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:11 +0000 (0:00:00.059) 0:01:47.135 ***** 2021-02-11T09:40:11.752 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-02-11T09:40:11.753 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-02-11T09:40:11.788 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:11 +0000 (0:00:00.217) 0:01:47.353 ***** 2021-02-11T09:40:11.814 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:11.847 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:11 +0000 (0:00:00.059) 0:01:47.412 ***** 2021-02-11T09:40:11.884 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-02-11T09:40:11.885 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-02-11T09:40:11.937 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:11 +0000 (0:00:00.090) 0:01:47.502 ***** 2021-02-11T09:40:11.966 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-02-11T09:40:11.967 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:12.004 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:12 +0000 (0:00:00.067) 0:01:47.569 ***** 2021-02-11T09:40:13.077 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-02-11T09:40:13.078 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-02-11T09:40:13.130 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-02-11T09:40:13.972 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-02-11T09:40:31.251 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:14 +0000 (0:00:02.004) 0:01:49.574 ***** ok: [smithi026.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} Thursday 11 February 2021 09:40:14 +0000 (0:00:00.095) 0:01:49.669 ***** TASK [testnode : Install python-apt] ******************************************* ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1613036134, "cache_updated": false, "changed": false} Thursday 11 February 2021 09:40:15 +0000 (0:00:00.923) 0:01:50.593 ***** ok: [smithi026.front.sepia.ceph.com] => {"cache_update_time": 1613036091, "cache_updated": false, "changed": false} Thursday 11 February 2021 09:40:15 +0000 (0:00:00.051) 0:01:50.644 ***** TASK [testnode : Add local apt repos.] ***************************************** Thursday 11 February 2021 09:40:15 +0000 (0:00:00.042) 0:01:50.687 ***** Thursday 11 February 2021 09:40:15 +0000 (0:00:00.042) 0:01:50.730 ***** TASK [testnode : Blast the apt cache.] ***************************************** [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.035905", "end": "2021-02-11 09:40:15.596044", "rc": 0, "start": "2021-02-11 09:40:15.560139", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 09:40:15 +0000 (0:00:00.167) 0:01:50.897 ***** changed: [smithi026.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.033036", "end": "2021-02-11 09:40:15.322006", "rc": 0, "start": "2021-02-11 09:40:15.288970", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 09:40:15 +0000 (0:00:00.042) 0:01:50.940 ***** TAS 2021-02-11T09:40:31.252 INFO:teuthology.task.ansible.out:K [testnode : Update apt cache.] ******************************************** ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1613036134, "cache_updated": false, "changed": false} Thursday 11 February 2021 09:40:18 +0000 (0:00:03.065) 0:01:54.006 ***** ok: [smithi026.front.sepia.ceph.com] => {"cache_update_time": 1613036091, "cache_updated": false, "changed": false} Thursday 11 February 2021 09:40:18 +0000 (0:00:00.051) 0:01:54.057 ***** TASK [testnode : Ensure packages are not present.] ***************************** changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 19 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(R 2021-02-11T09:40:31.254 INFO:teuthology.task.ansible.out:eading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137610 files and directories currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " lib 2021-02-11T09:40:31.255 INFO:teuthology.task.ansible.out:rados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 19 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137610 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} Thursday 11 February 2021 09:40:26 +0000 (0:00:08.375) 0:02:02.433 ***** changed: [smithi026.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-rom 2021-02-11T09:40:31.256 INFO:teuthology.task.ansible.out:s libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 19 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137610 files and directories currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Read 2021-02-11T09:40:31.258 INFO:teuthology.task.ansible.out:ing state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 19 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137610 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ..." 2021-02-11T09:40:31.259 INFO:teuthology.task.ansible.out:, "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} Thursday 11 February 2021 09:40:27 +0000 (0:00:00.807) 0:02:03.240 ***** TASK [testnode : Upgrade packages] ********************************************* ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1613036134, "cache_updated": false, "changed": false} Thursday 11 February 2021 09:40:27 +0000 (0:00:00.218) 0:02:03.458 ***** TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:40:27 +0000 (0:00:00.061) 0:02:03.519 ***** TASK [testnode : Upgrade packages] ********************************************* ok: [smithi026.front.sepia.ceph.com] => {"cache_update_time": 1613036091, "cache_updated": false, "changed": false} Thursday 11 February 2021 09:40:28 +0000 (0:00:00.671) 0:02:04.191 ***** TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:40:28 +0000 (0:00:00.059) 0:02:04.250 ***** 2021-02-11T09:40:36.343 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T09:40:36.344 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1613036134, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 19 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.35 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.35 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.35 [5194 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.35 [868 kB]\nFetched 10.4 MB in 1s (9823 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2021-02-11T09:40:36.345 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137468 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2021-02-11T09:40:36.346 INFO:teuthology.task.ansible.out:u1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 19 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.35 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.35 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.35 [5194 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.35 [868 kB]", "Fetched 10.4 MB in 1s (9823 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2021-02-11T09:40:36.348 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137468 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-02-11T09:40:36.380 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:36 +0000 (0:00:07.695) 0:02:11.945 ***** 2021-02-11T09:40:37.287 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-02-11T09:40:37.288 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1613036134, "cache_updated": false, "changed": false} 2021-02-11T09:40:37.323 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:37 +0000 (0:00:00.942) 0:02:12.888 ***** 2021-02-11T09:40:37.442 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-02-11T09:40:37.444 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-02-11T09:40:37.478 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:37 +0000 (0:00:00.155) 0:02:13.043 ***** 2021-02-11T09:40:37.504 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-02-11T09:40:37.505 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:37.540 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:37 +0000 (0:00:00.061) 0:02:13.105 ***** 2021-02-11T09:40:37.670 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-02-11T09:40:37.671 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-02-11T09:40:37.705 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:37 +0000 (0:00:00.165) 0:02:13.270 ***** 2021-02-11T09:40:37.785 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T09:40:37.786 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => {"cache_update_time": 1613036091, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 19 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.35 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.35 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.35 [5194 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.35 [868 kB]\nFetched 10.4 MB in 1s (9790 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2021-02-11T09:40:37.787 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137468 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2021-02-11T09:40:37.788 INFO:teuthology.task.ansible.out:u1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 19 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.35 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.35 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.35 [5194 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.35 [868 kB]", "Fetched 10.4 MB in 1s (9790 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2021-02-11T09:40:37.789 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137468 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-02-11T09:40:37.823 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:37 +0000 (0:00:00.117) 0:02:13.388 ***** 2021-02-11T09:40:38.030 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-02-11T09:40:38.031 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-02-11T09:40:38.067 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:38 +0000 (0:00:00.243) 0:02:13.632 ***** 2021-02-11T09:40:38.229 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-02-11T09:40:38.230 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-02-11T09:40:38.264 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:38 +0000 (0:00:00.197) 0:02:13.829 ***** 2021-02-11T09:40:38.384 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-02-11T09:40:38.386 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T09:40:38.420 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:38 +0000 (0:00:00.156) 0:02:13.986 ***** 2021-02-11T09:40:38.698 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-02-11T09:40:38.699 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 09:38:12 UTC", "ActiveEnterTimestampMonotonic": "22193632", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target remote-fs.target network.target basic.target nss-lookup.target -.mount system.slice systemd-journald.socket systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 09:38:12 UTC", "AssertTimestampMonotonic": "21647437", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 09:38:12 UTC", "ConditionTimestampMonotonic": "21647436", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-02-11T09:40:38.700 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1379", "ExecMainStartTimestamp": "Thu 2021-02-11 09:38:12 UTC", "ExecMainStartTimestampMonotonic": "22193579", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 09:38:12 UTC", "InactiveExitTimestampMonotonic": "21650216", "InvocationID": "2458dc41ea5741af90108a51130e68e2", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-02-11T09:40:38.701 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1379", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-02-11T09:40:38.703 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 09:38:12 UTC", "StateChangeTimestampMonotonic": "22193632", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 09:38:12 UTC", "WatchdogTimestampMonotonic": "22193629", "WatchdogUSec": "0"}} 2021-02-11T09:40:38.734 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:38 +0000 (0:00:00.313) 0:02:14.299 ***** 2021-02-11T09:40:38.739 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-02-11T09:40:38.740 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"cache_update_time": 1613036091, "cache_updated": false, "changed": false} 2021-02-11T09:40:38.775 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:38 +0000 (0:00:00.040) 0:02:14.340 ***** 2021-02-11T09:40:38.789 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-11T09:40:38.790 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:38.825 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:38 +0000 (0:00:00.049) 0:02:14.390 ***** 2021-02-11T09:40:38.856 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-11T09:40:38.858 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:38.892 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:38 +0000 (0:00:00.066) 0:02:14.457 ***** 2021-02-11T09:40:38.908 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-02-11T09:40:38.909 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-02-11T09:40:38.944 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:38 +0000 (0:00:00.051) 0:02:14.509 ***** 2021-02-11T09:40:38.950 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-11T09:40:38.951 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:38.986 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:38 +0000 (0:00:00.042) 0:02:14.551 ***** 2021-02-11T09:40:38.991 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-02-11T09:40:38.992 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:39.028 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:39 +0000 (0:00:00.041) 0:02:14.593 ***** 2021-02-11T09:40:39.034 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-11T09:40:39.036 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:39.072 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:39 +0000 (0:00:00.043) 0:02:14.637 ***** 2021-02-11T09:40:39.099 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-11T09:40:39.101 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:39.134 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:39 +0000 (0:00:00.062) 0:02:14.699 ***** 2021-02-11T09:40:39.160 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-02-11T09:40:39.161 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:39.202 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:39 +0000 (0:00:00.068) 0:02:14.768 ***** 2021-02-11T09:40:39.236 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:39 +0000 (0:00:00.033) 0:02:14.801 ***** 2021-02-11T09:40:39.262 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-11T09:40:39.264 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:39.298 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:39 +0000 (0:00:00.061) 0:02:14.863 ***** 2021-02-11T09:40:39.323 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-11T09:40:39.325 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:39.357 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:39 +0000 (0:00:00.058) 0:02:14.922 ***** 2021-02-11T09:40:39.383 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:40:39.384 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:39.419 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:39 +0000 (0:00:00.062) 0:02:14.984 ***** 2021-02-11T09:40:39.446 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:39.485 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:39 +0000 (0:00:00.066) 0:02:15.050 ***** 2021-02-11T09:40:39.501 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-02-11T09:40:39.501 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-02-11T09:40:39.536 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:39 +0000 (0:00:00.050) 0:02:15.101 ***** 2021-02-11T09:40:39.544 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T09:40:39.545 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:39.582 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:39 +0000 (0:00:00.045) 0:02:15.147 ***** 2021-02-11T09:40:39.609 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T09:40:39.610 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:39.645 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:39 +0000 (0:00:00.062) 0:02:15.210 ***** 2021-02-11T09:40:39.672 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-11T09:40:39.674 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:39.707 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:39 +0000 (0:00:00.062) 0:02:15.272 ***** 2021-02-11T09:40:39.714 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-02-11T09:40:39.715 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-02-11T09:40:39.751 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:39 +0000 (0:00:00.043) 0:02:15.316 ***** 2021-02-11T09:40:39.757 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T09:40:39.759 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:39.794 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:39 +0000 (0:00:00.042) 0:02:15.359 ***** 2021-02-11T09:40:39.820 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T09:40:39.822 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:39.856 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:39 +0000 (0:00:00.062) 0:02:15.421 ***** 2021-02-11T09:40:39.872 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-02-11T09:40:39.874 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T09:40:39.908 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:39 +0000 (0:00:00.051) 0:02:15.473 ***** 2021-02-11T09:40:39.917 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-11T09:40:39.918 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:39.954 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:39 +0000 (0:00:00.045) 0:02:15.519 ***** 2021-02-11T09:40:39.981 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-11T09:40:39.982 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.016 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.062) 0:02:15.582 ***** 2021-02-11T09:40:40.043 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-11T09:40:40.044 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.077 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.060) 0:02:15.642 ***** 2021-02-11T09:40:40.103 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-11T09:40:40.104 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.136 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.058) 0:02:15.701 ***** 2021-02-11T09:40:40.161 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T09:40:40.162 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.199 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.062) 0:02:15.764 ***** 2021-02-11T09:40:40.205 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-02-11T09:40:40.206 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 09:37:51 UTC", "ActiveEnterTimestampMonotonic": "19103554", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket remote-fs.target systemd-tmpfiles-setup.service sysinit.target system.slice nss-lookup.target -.mount basic.target network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 09:37:51 UTC", "AssertTimestampMonotonic": "18719255", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 09:37:51 UTC", "ConditionTimestampMonotonic": "18719254", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-02-11T09:40:40.208 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1351", "ExecMainStartTimestamp": "Thu 2021-02-11 09:37:51 UTC", "ExecMainStartTimestampMonotonic": "19103542", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 09:37:51 UTC", "InactiveExitTimestampMonotonic": "18719930", "InvocationID": "615d77252e354d678889cdef947af022", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-02-11T09:40:40.209 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1351", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-02-11T09:40:40.210 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 09:37:51 UTC", "StateChangeTimestampMonotonic": "19103554", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 09:37:51 UTC", "WatchdogTimestampMonotonic": "19103553", "WatchdogUSec": "0"}} 2021-02-11T09:40:40.243 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.044) 0:02:15.808 ***** 2021-02-11T09:40:40.250 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T09:40:40.252 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.285 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.042) 0:02:15.850 ***** 2021-02-11T09:40:40.291 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-11T09:40:40.292 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.327 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.041) 0:02:15.892 ***** 2021-02-11T09:40:40.333 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-11T09:40:40.334 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.369 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.041) 0:02:15.934 ***** 2021-02-11T09:40:40.375 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-11T09:40:40.376 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.411 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.041) 0:02:15.976 ***** 2021-02-11T09:40:40.416 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T09:40:40.417 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.452 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.041) 0:02:16.017 ***** 2021-02-11T09:40:40.457 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-11T09:40:40.458 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.492 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.040) 0:02:16.057 ***** 2021-02-11T09:40:40.499 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T09:40:40.501 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.534 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.042) 0:02:16.099 ***** 2021-02-11T09:40:40.540 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-11T09:40:40.541 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.576 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.042) 0:02:16.142 ***** 2021-02-11T09:40:40.582 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-11T09:40:40.584 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.618 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.041) 0:02:16.183 ***** 2021-02-11T09:40:40.623 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-11T09:40:40.624 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.660 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.042) 0:02:16.225 ***** 2021-02-11T09:40:40.666 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-02-11T09:40:40.667 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.700 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.040) 0:02:16.266 ***** 2021-02-11T09:40:40.706 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-11T09:40:40.708 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.742 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.041) 0:02:16.307 ***** 2021-02-11T09:40:40.768 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-11T09:40:40.769 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.802 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.060) 0:02:16.367 ***** 2021-02-11T09:40:40.835 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-02-11T09:40:40.835 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.869 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.066) 0:02:16.434 ***** 2021-02-11T09:40:40.901 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.032) 0:02:16.466 ***** 2021-02-11T09:40:40.907 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:40:40.909 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:40.944 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:40 +0000 (0:00:00.042) 0:02:16.509 ***** 2021-02-11T09:40:40.973 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:41.009 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.064) 0:02:16.574 ***** 2021-02-11T09:40:41.036 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T09:40:41.037 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:41.072 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.062) 0:02:16.637 ***** 2021-02-11T09:40:41.099 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T09:40:41.100 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:41.135 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.063) 0:02:16.700 ***** 2021-02-11T09:40:41.161 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-11T09:40:41.162 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:41.195 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.060) 0:02:16.761 ***** 2021-02-11T09:40:41.201 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-11T09:40:41.201 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-02-11T09:40:41.237 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.042) 0:02:16.803 ***** 2021-02-11T09:40:41.242 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T09:40:41.244 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:41.279 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.041) 0:02:16.844 ***** 2021-02-11T09:40:41.285 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-11T09:40:41.287 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:41.323 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.044) 0:02:16.888 ***** 2021-02-11T09:40:41.328 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T09:40:41.330 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:41.366 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.042) 0:02:16.931 ***** 2021-02-11T09:40:41.392 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:41.427 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.061) 0:02:16.993 ***** 2021-02-11T09:40:41.454 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-11T09:40:41.455 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:41.493 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.065) 0:02:17.058 ***** 2021-02-11T09:40:41.521 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:41.557 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.064) 0:02:17.122 ***** 2021-02-11T09:40:41.584 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-11T09:40:41.585 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:41.618 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.061) 0:02:17.183 ***** 2021-02-11T09:40:41.644 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T09:40:41.646 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Upload weekly kernel-clean crontab.] ************************** ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-02-11T09:40:41.681 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.062) 0:02:17.246 ***** 2021-02-11T09:40:41.710 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.029) 0:02:17.276 ***** 2021-02-11T09:40:41.717 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T09:40:41.718 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:41.751 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.040) 0:02:17.316 ***** 2021-02-11T09:40:41.777 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-11T09:40:41.778 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:41.810 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.059) 0:02:17.375 ***** 2021-02-11T09:40:41.836 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T09:40:41.838 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:41.870 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.059) 0:02:17.435 ***** 2021-02-11T09:40:41.895 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T09:40:41.896 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:41.931 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.060) 0:02:17.496 ***** 2021-02-11T09:40:41.959 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:41.998 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:41 +0000 (0:00:00.067) 0:02:17.563 ***** 2021-02-11T09:40:42.015 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-02-11T09:40:42.016 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-02-11T09:40:42.049 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:42 +0000 (0:00:00.050) 0:02:17.614 ***** 2021-02-11T09:40:42.054 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-02-11T09:40:42.054 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:42.094 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:42 +0000 (0:00:00.045) 0:02:17.659 ***** 2021-02-11T09:40:42.224 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-02-11T09:40:42.226 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-02-11T09:40:42.258 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:42 +0000 (0:00:00.163) 0:02:17.823 ***** 2021-02-11T09:40:42.346 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-02-11T09:40:42.348 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-02-11T09:40:42.383 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:42 +0000 (0:00:00.125) 0:02:17.948 ***** 2021-02-11T09:40:42.523 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-02-11T09:40:42.525 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T09:40:42.557 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:42 +0000 (0:00:00.174) 0:02:18.123 ***** 2021-02-11T09:40:42.563 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-11T09:40:42.563 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-02-11T09:40:42.596 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:42 +0000 (0:00:00.039) 0:02:18.162 ***** 2021-02-11T09:40:42.601 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-02-11T09:40:42.602 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:42.634 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:42 +0000 (0:00:00.037) 0:02:18.199 ***** 2021-02-11T09:40:42.639 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-11T09:40:42.640 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:42.673 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:42 +0000 (0:00:00.038) 0:02:18.238 ***** 2021-02-11T09:40:42.678 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-02-11T09:40:42.679 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:42.712 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:42 +0000 (0:00:00.039) 0:02:18.277 ***** 2021-02-11T09:40:42.737 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-02-11T09:40:42.739 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:42.771 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:42 +0000 (0:00:00.059) 0:02:18.336 ***** 2021-02-11T09:40:42.797 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-02-11T09:40:42.798 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:42.830 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:42 +0000 (0:00:00.058) 0:02:18.395 ***** 2021-02-11T09:40:42.855 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-02-11T09:40:42.856 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:42.889 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:42 +0000 (0:00:00.058) 0:02:18.454 ***** 2021-02-11T09:40:42.914 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:40:42.915 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-02-11T09:40:42.948 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:42 +0000 (0:00:00.058) 0:02:18.513 ***** 2021-02-11T09:40:42.973 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-02-11T09:40:42.974 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-02-11T09:40:43.010 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:43 +0000 (0:00:00.062) 0:02:18.575 ***** 2021-02-11T09:40:43.016 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-02-11T09:40:43.017 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:43.018 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:43.018 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:43.020 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:43.030 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:43.041 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-02-11T09:40:43.042 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:43.052 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:43.052 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:43.063 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:43.063 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:43.074 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:43.085 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:43.085 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:43.123 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:43 +0000 (0:00:00.113) 0:02:18.689 ***** 2021-02-11T09:40:43.311 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-02-11T09:40:43.312 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003144", "end": "2021-02-11 09:40:43.284310", "rc": 0, "start": "2021-02-11 09:40:43.281166", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock"]} TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-02-11T09:40:43.344 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:43 +0000 (0:00:00.220) 0:02:18.909 ***** 2021-02-11T09:40:43.379 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:43 +0000 (0:00:00.034) 0:02:18.944 ***** 2021-02-11T09:40:43.404 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-02-11T09:40:43.406 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:43.442 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:43 +0000 (0:00:00.063) 0:02:19.007 ***** 2021-02-11T09:40:43.467 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-02-11T09:40:43.468 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:43.500 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:43 +0000 (0:00:00.058) 0:02:19.065 ***** 2021-02-11T09:40:43.525 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-02-11T09:40:43.526 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:43.558 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:43 +0000 (0:00:00.057) 0:02:19.123 ***** 2021-02-11T09:40:43.635 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-02-11T09:40:43.636 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-02-11T09:40:43.678 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:43 +0000 (0:00:00.120) 0:02:19.243 ***** 2021-02-11T09:40:43.810 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-02-11T09:40:43.811 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T09:40:43.845 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:43 +0000 (0:00:00.167) 0:02:19.410 ***** 2021-02-11T09:40:43.872 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-02-11T09:40:43.873 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:43.907 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:43 +0000 (0:00:00.061) 0:02:19.472 ***** 2021-02-11T09:40:43.934 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-02-11T09:40:43.935 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:43.971 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:43 +0000 (0:00:00.063) 0:02:19.536 ***** 2021-02-11T09:40:43.997 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-02-11T09:40:43.998 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:44.031 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:44 +0000 (0:00:00.060) 0:02:19.596 ***** 2021-02-11T09:40:44.057 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-02-11T09:40:44.058 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:44.094 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:44 +0000 (0:00:00.062) 0:02:19.659 ***** 2021-02-11T09:40:44.120 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-02-11T09:40:44.121 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:44.153 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:44 +0000 (0:00:00.058) 0:02:19.718 ***** 2021-02-11T09:40:44.179 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:40:44.180 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-02-11T09:40:44.213 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:44 +0000 (0:00:00.060) 0:02:19.778 ***** 2021-02-11T09:40:44.259 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-02-11T09:40:44.261 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi026.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:44.271 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:44.281 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:44.282 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:44.293 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:44.303 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-02-11T09:40:44.314 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:44.315 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:44.326 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:44.326 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:44.337 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:44.347 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:44.348 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:44.359 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-02-11T09:40:59.896 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:44 +0000 (0:00:00.183) 0:02:19.961 ***** TASK [testnode : Make sure apt dependencies are installed] ********************* ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1613036134, "cache_updated": false, "changed": false} Thursday 11 February 2021 09:40:44 +0000 (0:00:00.093) 0:02:20.055 ***** TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:40:44 +0000 (0:00:00.059) 0:02:20.115 ***** TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi026.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003063", "end": "2021-02-11 09:40:44.550629", "rc": 0, "start": "2021-02-11 09:40:44.547566", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock"]} Thursday 11 February 2021 09:40:44 +0000 (0:00:00.067) 0:02:20.182 ***** TASK [testnode : Set root disk] ************************************************ ok: [smithi133.front.sepia.ceph.com] => (item={'block_used': 13754871, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340237, 'size_available': 927089610752, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 2263402 2021-02-11T09:40:59.897 INFO:teuthology.task.ansible.out:37, "block_size": 4096, "block_total": 240095108, "block_used": 13754871, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089610752, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}} Thursday 11 February 2021 09:40:44 +0000 (0:00:00.042) 0:02:20.225 ***** TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:40:44 +0000 (0:00:00.044) 0:02:20.270 ***** TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:40:44 +0000 (0:00:00.060) 0:02:20.331 ***** TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:40:44 +0000 (0:00:00.059) 0:02:20.390 ***** TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009456", "end": "2021-02-11 09:40:45.084924", "rc": 0, "start": "2021-02-11 09:40:45.075468", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 09:40:44 +0000 (0:00:00.040) 0:02:20.430 ***** TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi133.front.sepia.ceph.com] => (item={'block_used': 13754871, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_t 2021-02-11T09:40:59.898 INFO:teuthology.task.ansible.out:otal': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340237, 'size_available': 927089610752, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340237, "block_size": 4096, "block_total": 240095108, "block_used": 13754871, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089610752, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:40:44 +0000 (0:00:00.087) 0:02:20.517 ***** TASK [testnode : Wipe filesystems on non-root partitions] ********************** Thursday 11 February 2021 09:40:45 +0000 (0:00:00.061) 0:02:20.579 ***** TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": nu 2021-02-11T09:40:59.900 INFO:teuthology.task.ansible.out:ll, "virtual": 1}}, "skip_reason": "Conditional result was False"} TASK [testnode : Make sure apt dependencies are installed] ********************* ok: [smithi026.front.sepia.ceph.com] => {"cache_update_time": 1613036091, "cache_updated": false, "changed": false} Thursday 11 February 2021 09:40:45 +0000 (0:00:00.758) 0:02:21.337 ***** TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:40:45 +0000 (0:00:00.059) 0:02:21.397 ***** TASK [testnode : Set root disk] ************************************************ ok: [smithi026.front.sepia.ceph.com] => (item={'block_used': 13756931, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226338177, 'size_available': 927081172992, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226338177, "block_size": 4096, "block_total": 240095108, "block_used": 13756931, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927081172992, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}} Thursday 11 February 2021 09:40:45 +0000 (0:00:00.069) 0:02:21.466 ***** TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi026.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009179", "e 2021-02-11T09:40:59.901 INFO:teuthology.task.ansible.out:nd": "2021-02-11 09:40:46.022772", "rc": 0, "start": "2021-02-11 09:40:46.013593", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 09:40:46 +0000 (0:00:00.163) 0:02:21.630 ***** TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi026.front.sepia.ceph.com] => (item={'block_used': 13756931, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226338177, 'size_available': 927081172992, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226338177, "block_size": 4096, "block_total": 240095108, "block_used": 13756931, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927081172992, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:40:46 +0000 (0:00:00.069) 0:02:21.700 ***** TASK [testnode : Wipe filesystems on non-root partitions] ********************** Thursday 11 February 2021 09:40:46 +0000 (0:00:00.058) 0:02:21.759 ***** TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'hold 2021-02-11T09:40:59.902 INFO:teuthology.task.ansible.out:ers': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN5KUjGoKTfabX9l9VoHAoa4UmOuQdQVnkb'], 'uuids': ['5ec27e74-c053-45e5-bdd9-e381a7813185']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:02.115687", "end": "2021-02-11 09:40:47.583983", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN5KUjGoKTfabX9l9VoHAoa4UmOuQdQVnkb"], "labels": [], "masters": [], "uuids": ["5ec27e74-c053-45e5-bdd9-e381a7813185"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:40:45.468296", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be 2021-02-11T09:40:59.904 INFO:teuthology.task.ansible.out:used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": 2021-02-11T09:40:59.905 INFO:teuthology.task.ansible.out:null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKSe1pbuDiFDz5Jr9AbbmcutNPWHw4jGCF6'], 'uuids': ['5301c9dd-d8b1-494e-a1fc-7753b74ed855']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.977900", "end": "2021-02-11 09:40:48.326180", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKSe1pbuDiFDz5Jr9AbbmcutNPWHw4jGCF6"], "labels": [], "masters": [], "uuids": ["5301c9dd-d8b1-494e-a1fc-7753b74ed855"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 2021-02-11T09:40:59.905 INFO:teuthology.task.ansible.out: 0, "start": "2021-02-11 09:40:46.348280", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning 2021-02-11T09:40:59.906 INFO:teuthology.task.ansible.out:: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T09:40:59.908 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:59.908 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:59.909 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN5H1HKSp8TEezFvrHbJYSnN8qlXWZeRV7E'], 'uuids': ['8d00be60-6ca5-426d-98c8-258955702cc4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.115841", "end": "2021-02-11 09:40:48.818441", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN5H1HKSp8TEezFvrHbJYSnN8qlXWZeRV7E"], "labels": [], "masters": [], "uuids": ["8d00be60-6ca5-426d-98c8-258955702cc4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:40:47.702600", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T09:40:59.910 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XPD9', 'wwn-0x5000c50092928ee8'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XPD9-part1', 'wwn-0x5000c50092928ee8-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092928ee8', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XPD9', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XPD9", "wwn-0x5000c50092928ee8"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XPD9-part1", "wwn-0x5000c50092928ee8-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XPD9", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092928ee8"}}, "skip_reason": 2021-02-11T09:40:59.912 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-11T09:40:59.912 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKSWThH6EisXotr4OCOaVBXllMZ1JlsKBys'], 'uuids': ['6cbed70a-7244-43dd-bebe-a39757af84c2']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.090003", "end": "2021-02-11 09:40:49.555043", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKSWThH6EisXotr4OCOaVBXllMZ1JlsKBys"], "labels": [], "masters": [], "uuids": ["6cbed70a-7244-43dd-bebe-a39757af84c2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:40:48.465040", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T09:40:59.913 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4H3KL', 'wwn-0x5000c500871b4821'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4H3KL-part1', 'wwn-0x5000c500871b4821-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b4821', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4H3KL', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H3KL", "wwn-0x5000c500871b4821"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H3KL-part1", "wwn-0x5000c500871b4821-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4H3KL", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b4821"}}, "skip_reason": 2021-02-11T09:40:59.914 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-11T09:40:59.916 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-O2IJs6-B1ll-Qufm-YIuj-HBJt-guOK-GszoKw', 'nvme-INTEL_SSDPEDMD400G4_PHFT620401AQ400BGN', 'nvme-nvme.8086-50484654363230343031415134303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.203882", "end": "2021-02-11 09:40:50.156365", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-O2IJs6-B1ll-Qufm-YIuj-HBJt-guOK-GszoKw", "nvme-INTEL_SSDPEDMD400G4_PHFT620401AQ400BGN", "nvme-nvme.8086-50484654363230343031415134303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:40:48.952483", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-02-11T09:40:59.917 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_re 2021-02-11T09:40:59.918 INFO:teuthology.task.ansible.out:ason": "Conditional result was False"} changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-sHL1Px-YlGn-mfBL-L8Nc-fbo0-EXmU-JBf3Uq', 'nvme-INTEL_SSDPEDMD400G4_CVFT5331003G400BGN', 'nvme-nvme.8086-43564654353333313030334734303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.161693", "end": "2021-02-11 09:40:50.844613", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-sHL1Px-YlGn-mfBL-L8Nc-fbo0-EXmU-JBf3Uq", "nvme-INTEL_SSDPEDMD400G4_CVFT5331003G400BGN", "nvme-nvme.8086-43564654353333313030334734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:40:49.682920", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT e 2021-02-11T09:40:59.919 INFO:teuthology.task.ansible.out:ntries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:59.921 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:59.922 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN501DtEWt1Z4HUcKOU6sMq5eXlmgYGrOsA'], 'uuids': ['1b2dbb6c-dbab-419c-a711-9dd90d48d9c4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.164547", "end": "2021-02-11 09:40:51.443372", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN501DtEWt1Z4HUcKOU6sMq5eXlmgYGrOsA"], "labels": [], "masters": [], "uuids": ["1b2dbb6c-dbab-419c-a711-9dd90d48d9c4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:40:50.278825", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T09:40:59.923 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:59.925 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKSgmBI9fs4l5FbnaDstU2tKidinHd6q11g'], 'uuids': ['8b664591-e6d3-40a4-ac70-7a9be2a2c3c5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.136281", "end": "2021-02-11 09:40:52.123922", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKSgmBI9fs4l5FbnaDstU2tKidinHd6q11g"], "labels": [], "masters": [], "uuids": ["8b664591-e6d3-40a4-ac70-7a9be2a2c3c5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:40:50.987641", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T09:40:59.926 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:59.927 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:59.929 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN5gtfa4TcDSNQMGPPWBofN3473OsJ20nyR'], 'uuids': ['e994103e-b063-4fa9-8e46-a76cd2e20c4a']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.108402", "end": "2021-02-11 09:40:52.684550", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN5gtfa4TcDSNQMGPPWBofN3473OsJ20nyR"], "labels": [], "masters": [], "uuids": ["e994103e-b063-4fa9-8e46-a76cd2e20c4a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:40:51.576148", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi133.front.sepia.ceph.com] => 2021-02-11T09:40:59.930 INFO:teuthology.task.ansible.out:(item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKSGsE4ij0Vi0aEMRkeYZ6RG6wSzFaNc3bi'], 'uuids': ['4218f4fb-acfc-4553-8b0b-38fcfc8ada66']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.119973", "end": "2021-02-11 09:40:53.388942", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKSGsE4ij0Vi0aEMRkeYZ6RG6wSzFaNc3bi"], "labels": [], "masters": [], "uuids": ["4218f4fb-acfc-4553-8b0b-38fcfc8ada66"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, 2021-02-11T09:40:59.931 INFO:teuthology.task.ansible.out: "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:40:52.268969", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:40:59.932 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN592inTC440yLRjLln5zQcRK6fxL048Kzv'], 'uuids': ['a0908d27-c15b-43dc-8e6a-76d57ac4da0b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.126458", "end": "2021-02-11 09:40:53.925858", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN592inTC440yLRjLln5zQcRK6fxL048Kzv"], "labels": [], "masters": [], "uuids": ["a0908d27-c15b-43dc-8e6a-76d57ac4da0b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:40:52.799400", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T09:40:59.943 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKS6VN3EcG76qz0Ex9dgZvXQg0kBEryZAZU'], 'uuids': ['9bbfa9df-8ccf-4ba5-871e-0857f846d86c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.100749", "end": "2021-02-11 09:40:54.614604", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKS6VN3EcG76qz0Ex9dgZvXQg0kBEryZAZU"], "labels": [], "masters": [], "uuids": ["9bbfa9df-8ccf-4ba5-871e-0857f846d86c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:40:53.513855", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T09:40:59.989 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:40:59 +0000 (0:00:13.795) 0:02:35.554 ***** 2021-02-11T09:41:00.024 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:00 +0000 (0:00:00.034) 0:02:35.589 ***** 2021-02-11T09:41:00.044 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T09:41:00.044 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:00.243 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:00.493 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKSe1pbuDiFDz5Jr9AbbmcutNPWHw4jGCF6'], 'uuids': ['5301c9dd-d8b1-494e-a1fc-7753b74ed855']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.154708", "end": "2021-02-11 09:41:00.473524", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKSe1pbuDiFDz5Jr9AbbmcutNPWHw4jGCF6"], "labels": [], "masters": [], "uuids": ["5301c9dd-d8b1-494e-a1fc-7753b74ed855"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:41:00.318816", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.150009 s, 769 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.150009 s, 769 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:00.505 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:00.506 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:00.528 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN5KUjGoKTfabX9l9VoHAoa4UmOuQdQVnkb'], 'uuids': ['5ec27e74-c053-45e5-bdd9-e381a7813185']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.154386", "end": "2021-02-11 09:41:00.804813", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN5KUjGoKTfabX9l9VoHAoa4UmOuQdQVnkb"], "labels": [], "masters": [], "uuids": ["5ec27e74-c053-45e5-bdd9-e381a7813185"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:41:00.650427", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.1501 s, 768 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.1501 s, 768 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:00.539 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:00.550 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:00.800 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKSWThH6EisXotr4OCOaVBXllMZ1JlsKBys'], 'uuids': ['6cbed70a-7244-43dd-bebe-a39757af84c2']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.163597", "end": "2021-02-11 09:41:00.779640", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKSWThH6EisXotr4OCOaVBXllMZ1JlsKBys"], "labels": [], "masters": [], "uuids": ["6cbed70a-7244-43dd-bebe-a39757af84c2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:41:00.616043", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.160535 s, 718 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.160535 s, 718 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:00.801 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4H3KL', 'wwn-0x5000c500871b4821'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4H3KL-part1', 'wwn-0x5000c500871b4821-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b4821', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4H3KL', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H3KL", "wwn-0x5000c500871b4821"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H3KL-part1", "wwn-0x5000c500871b4821-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4H3KL", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b4821"}}, "skip_reason": 2021-02-11T09:41:00.803 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-11T09:41:00.823 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN5H1HKSp8TEezFvrHbJYSnN8qlXWZeRV7E'], 'uuids': ['8d00be60-6ca5-426d-98c8-258955702cc4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.154332", "end": "2021-02-11 09:41:01.097383", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN5H1HKSp8TEezFvrHbJYSnN8qlXWZeRV7E"], "labels": [], "masters": [], "uuids": ["8d00be60-6ca5-426d-98c8-258955702cc4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:41:00.943051", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149867 s, 770 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149867 s, 770 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:00.825 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XPD9', 'wwn-0x5000c50092928ee8'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XPD9-part1', 'wwn-0x5000c50092928ee8-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092928ee8', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XPD9', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XPD9", "wwn-0x5000c50092928ee8"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XPD9-part1", "wwn-0x5000c50092928ee8-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XPD9", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092928ee8"}}, "skip_reason": 2021-02-11T09:41:00.826 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-11T09:41:00.971 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-sHL1Px-YlGn-mfBL-L8Nc-fbo0-EXmU-JBf3Uq', 'nvme-INTEL_SSDPEDMD400G4_CVFT5331003G400BGN', 'nvme-nvme.8086-43564654353333313030334734303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.048856", "end": "2021-02-11 09:41:00.955366", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-sHL1Px-YlGn-mfBL-L8Nc-fbo0-EXmU-JBf3Uq", "nvme-INTEL_SSDPEDMD400G4_CVFT5331003G400BGN", "nvme-nvme.8086-43564654353333313030334734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:41:00.906510", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.044122 s, 2.6 GB/s", "stde 2021-02-11T09:41:00.972 INFO:teuthology.task.ansible.out:rr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.044122 s, 2.6 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:00.982 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:00.983 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:01.005 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-O2IJs6-B1ll-Qufm-YIuj-HBJt-guOK-GszoKw', 'nvme-INTEL_SSDPEDMD400G4_PHFT620401AQ400BGN', 'nvme-nvme.8086-50484654363230343031415134303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.048510", "end": "2021-02-11 09:41:01.277328", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-O2IJs6-B1ll-Qufm-YIuj-HBJt-guOK-GszoKw", "nvme-INTEL_SSDPEDMD400G4_PHFT620401AQ400BGN", "nvme-nvme.8086-50484654363230343031415134303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:41:01.228818", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0438639 s, 2.6 GB/s", "std 2021-02-11T09:41:01.007 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0438639 s, 2.6 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:01.008 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:01.018 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:01.246 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKSgmBI9fs4l5FbnaDstU2tKidinHd6q11g'], 'uuids': ['8b664591-e6d3-40a4-ac70-7a9be2a2c3c5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.145161", "end": "2021-02-11 09:41:01.233947", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKSgmBI9fs4l5FbnaDstU2tKidinHd6q11g"], "labels": [], "masters": [], "uuids": ["8b664591-e6d3-40a4-ac70-7a9be2a2c3c5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:41:01.088786", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142802 s, 808 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142802 s, 808 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:01.258 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:01.269 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:01.301 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN501DtEWt1Z4HUcKOU6sMq5eXlmgYGrOsA'], 'uuids': ['1b2dbb6c-dbab-419c-a711-9dd90d48d9c4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.157947", "end": "2021-02-11 09:41:01.573914", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN501DtEWt1Z4HUcKOU6sMq5eXlmgYGrOsA"], "labels": [], "masters": [], "uuids": ["1b2dbb6c-dbab-419c-a711-9dd90d48d9c4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:41:01.415967", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.15505 s, 744 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.15505 s, 744 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:01.303 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:01.314 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:01.542 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKSGsE4ij0Vi0aEMRkeYZ6RG6wSzFaNc3bi'], 'uuids': ['4218f4fb-acfc-4553-8b0b-38fcfc8ada66']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.156088", "end": "2021-02-11 09:41:01.528564", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKSGsE4ij0Vi0aEMRkeYZ6RG6wSzFaNc3bi"], "labels": [], "masters": [], "uuids": ["4218f4fb-acfc-4553-8b0b-38fcfc8ada66"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:41:01.372476", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151294 s, 762 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151294 s, 762 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:01.554 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:01.587 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN5gtfa4TcDSNQMGPPWBofN3473OsJ20nyR'], 'uuids': ['e994103e-b063-4fa9-8e46-a76cd2e20c4a']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.157425", "end": "2021-02-11 09:41:01.853328", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN5gtfa4TcDSNQMGPPWBofN3473OsJ20nyR"], "labels": [], "masters": [], "uuids": ["e994103e-b063-4fa9-8e46-a76cd2e20c4a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:41:01.695903", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154611 s, 746 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154611 s, 746 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:01.598 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:01.827 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKS6VN3EcG76qz0Ex9dgZvXQg0kBEryZAZU'], 'uuids': ['9bbfa9df-8ccf-4ba5-871e-0857f846d86c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.158312", "end": "2021-02-11 09:41:01.813914", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-ssWvj866verA2Wak6409lVeMhRficEKS6VN3EcG76qz0Ex9dgZvXQg0kBEryZAZU"], "labels": [], "masters": [], "uuids": ["9bbfa9df-8ccf-4ba5-871e-0857f846d86c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:41:01.655602", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153454 s, 752 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153454 s, 752 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:01.873 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:01 +0000 (0:00:01.849) 0:02:37.438 ***** 2021-02-11T09:41:01.880 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN592inTC440yLRjLln5zQcRK6fxL048Kzv'], 'uuids': ['a0908d27-c15b-43dc-8e6a-76d57ac4da0b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.152287", "end": "2021-02-11 09:41:02.134701", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-sWt93Cq4MyR036EZeuOFB3jeZx9UnNN592inTC440yLRjLln5zQcRK6fxL048Kzv"], "labels": [], "masters": [], "uuids": ["a0908d27-c15b-43dc-8e6a-76d57ac4da0b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:41:01.982414", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148365 s, 777 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148365 s, 777 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:01.927 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:01 +0000 (0:00:00.053) 0:02:37.492 ***** 2021-02-11T09:41:02.144 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-02-11T09:41:02.145 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.136055", "end": "2021-02-11 09:41:02.124508", "rc": 0, "start": "2021-02-11 09:41:01.988453", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:02.179 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:02 +0000 (0:00:00.251) 0:02:37.744 ***** 2021-02-11T09:41:02.206 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.128884", "end": "2021-02-11 09:41:02.468482", "rc": 0, "start": "2021-02-11 09:41:02.339598", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:02.242 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:02 +0000 (0:00:00.063) 0:02:37.807 ***** 2021-02-11T09:41:02.361 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-02-11T09:41:02.362 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.053534", "end": "2021-02-11 09:41:02.352250", "rc": 0, "start": "2021-02-11 09:41:02.298716", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:02.396 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:02 +0000 (0:00:00.154) 0:02:37.961 ***** 2021-02-11T09:41:02.423 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-02-11T09:41:02.460 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:02 +0000 (0:00:00.063) 0:02:38.025 ***** 2021-02-11T09:41:02.466 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-02-11T09:41:02.467 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.064507", "end": "2021-02-11 09:41:02.705647", "rc": 0, "start": "2021-02-11 09:41:02.641140", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:02.504 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:02 +0000 (0:00:00.043) 0:02:38.069 ***** 2021-02-11T09:41:02.509 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-02-11T09:41:02.510 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:02.631 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:02 +0000 (0:00:00.126) 0:02:38.196 ***** 2021-02-11T09:41:02.638 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-02-11T09:41:02.675 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:02 +0000 (0:00:00.044) 0:02:38.240 ***** 2021-02-11T09:41:02.681 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-02-11T09:41:02.681 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:02.717 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:02 +0000 (0:00:00.041) 0:02:38.282 ***** 2021-02-11T09:41:02.723 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-02-11T09:41:02.723 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:02.758 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:02 +0000 (0:00:00.041) 0:02:38.324 ***** 2021-02-11T09:41:02.763 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-11T09:41:02.765 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:02.801 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:02 +0000 (0:00:00.042) 0:02:38.366 ***** 2021-02-11T09:41:02.808 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-02-11T09:41:02.809 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:02.842 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:02 +0000 (0:00:00.041) 0:02:38.408 ***** 2021-02-11T09:41:02.848 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-02-11T09:41:02.848 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:02.883 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:02 +0000 (0:00:00.041) 0:02:38.449 ***** 2021-02-11T09:41:02.889 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-11T09:41:02.890 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:02.929 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:02 +0000 (0:00:00.045) 0:02:38.495 ***** 2021-02-11T09:41:02.956 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-02-11T09:41:02.957 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:02.991 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:02 +0000 (0:00:00.061) 0:02:38.556 ***** 2021-02-11T09:41:03.008 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-02-11T09:41:03.009 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T09:41:03.045 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:03 +0000 (0:00:00.053) 0:02:38.610 ***** 2021-02-11T09:41:03.083 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-11T09:41:03.084 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'block_used': 13756931, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226338177, 'size_available': 927081172992, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226338177, "block_size": 4096, "block_total": 240095108, "block_used": 13756931, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927081172992, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:03.119 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:03 +0000 (0:00:00.074) 0:02:38.685 ***** 2021-02-11T09:41:03.126 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-02-11T09:41:03.127 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T09:41:03.163 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:03 +0000 (0:00:00.043) 0:02:38.729 ***** 2021-02-11T09:41:03.169 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-02-11T09:41:03.169 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:03.205 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:03 +0000 (0:00:00.041) 0:02:38.771 ***** 2021-02-11T09:41:03.219 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-11T09:41:03.219 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'block_used': 13754871, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340237, 'size_available': 927089610752, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340237, "block_size": 4096, "block_total": 240095108, "block_used": 13754871, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089610752, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:03.255 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:03 +0000 (0:00:00.049) 0:02:38.820 ***** 2021-02-11T09:41:03.261 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:41:03.261 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:03.296 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:03 +0000 (0:00:00.041) 0:02:38.861 ***** 2021-02-11T09:41:03.302 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-02-11T09:41:03.303 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:03.338 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:03 +0000 (0:00:00.041) 0:02:38.903 ***** 2021-02-11T09:41:03.344 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:41:03.345 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:03.379 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:03 +0000 (0:00:00.041) 0:02:38.944 ***** 2021-02-11T09:41:03.384 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:41:03.385 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:03.421 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:03 +0000 (0:00:00.041) 0:02:38.986 ***** 2021-02-11T09:41:03.427 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-02-11T09:41:03.427 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:03.462 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:03 +0000 (0:00:00.041) 0:02:39.027 ***** 2021-02-11T09:41:03.469 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:41:03.469 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:03.504 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:03 +0000 (0:00:00.041) 0:02:39.069 ***** 2021-02-11T09:41:03.509 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-02-11T09:41:03.510 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:03.543 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:03 +0000 (0:00:00.038) 0:02:39.108 ***** 2021-02-11T09:41:03.548 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-02-11T09:41:03.549 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:03.581 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:03 +0000 (0:00:00.038) 0:02:39.146 ***** 2021-02-11T09:41:03.585 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-02-11T09:41:03.587 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:03.619 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:03 +0000 (0:00:00.037) 0:02:39.184 ***** 2021-02-11T09:41:03.623 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-02-11T09:41:03.625 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:03.659 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:03 +0000 (0:00:00.039) 0:02:39.224 ***** 2021-02-11T09:41:03.685 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-02-11T09:41:03.686 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:03.718 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:03 +0000 (0:00:00.059) 0:02:39.284 ***** 2021-02-11T09:41:04.044 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-02-11T09:41:04.046 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-02-11T09:41:04.083 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:04 +0000 (0:00:00.364) 0:02:39.648 ***** 2021-02-11T09:41:04.089 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-02-11T09:41:04.129 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:04 +0000 (0:00:00.045) 0:02:39.694 ***** 2021-02-11T09:41:04.500 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-11T09:41:04.501 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:41:04.511 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:41:04.728 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:41:04.760 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:41:04.938 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:41:05.042 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:41:05.157 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:41:05.283 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:41:05.367 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:41:05.405 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:05 +0000 (0:00:01.276) 0:02:40.971 ***** 2021-02-11T09:41:05.516 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:41:05.554 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:05 +0000 (0:00:00.148) 0:02:41.119 ***** 2021-02-11T09:41:05.560 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-11T09:41:05.562 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-11T09:41:05.598 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:05 +0000 (0:00:00.044) 0:02:41.163 ***** 2021-02-11T09:41:05.677 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-11T09:41:05.720 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:05 +0000 (0:00:00.122) 0:02:41.285 ***** 2021-02-11T09:41:05.741 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-11T09:41:05.742 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T09:41:05.859 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T09:41:05.871 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T09:41:05.966 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T09:41:05.977 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T09:41:06.083 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T09:41:06.085 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:06.087 INFO:teuthology.task.ansible.out: 2021-02-11T09:41:06.093 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T09:41:06.094 INFO:teuthology.task.ansible.out: 2021-02-11T09:41:06.133 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:06 +0000 (0:00:00.412) 0:02:41.698 ***** 2021-02-11T09:41:06.160 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-02-11T09:41:06.161 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:06.192 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:06 +0000 (0:00:00.059) 0:02:41.757 ***** 2021-02-11T09:41:06.207 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-11T09:41:06.209 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T09:41:06.219 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:06.221 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-02-11T09:41:06.223 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:06.258 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:06 +0000 (0:00:00.065) 0:02:41.823 ***** 2021-02-11T09:41:06.290 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:06 +0000 (0:00:00.032) 0:02:41.855 ***** 2021-02-11T09:41:06.297 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-02-11T09:41:06.298 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:06.332 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:06 +0000 (0:00:00.042) 0:02:41.898 ***** 2021-02-11T09:41:06.359 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-02-11T09:41:06.360 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:06.392 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:06 +0000 (0:00:00.059) 0:02:41.957 ***** 2021-02-11T09:41:06.428 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-02-11T09:41:06.429 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:41:06.465 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:06 +0000 (0:00:00.072) 0:02:42.030 ***** 2021-02-11T09:41:06.501 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-02-11T09:41:06.503 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-02-11T09:41:06.537 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:06 +0000 (0:00:00.072) 0:02:42.102 ***** 2021-02-11T09:41:06.542 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-02-11T09:41:06.542 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:41:06.578 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:06 +0000 (0:00:00.040) 0:02:42.143 ***** 2021-02-11T09:41:06.615 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi026.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-02-11T09:41:06.649 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:06 +0000 (0:00:00.070) 0:02:42.214 ***** 2021-02-11T09:41:06.873 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-02-11T09:41:06.874 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true} 2021-02-11T09:41:06.908 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:06 +0000 (0:00:00.259) 0:02:42.474 ***** 2021-02-11T09:41:07.235 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-02-11T09:41:07.237 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-02-11T09:41:07.271 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:07 +0000 (0:00:00.362) 0:02:42.836 ***** 2021-02-11T09:41:07.297 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-02-11T09:41:07.298 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:07.334 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:07 +0000 (0:00:00.062) 0:02:42.899 ***** 2021-02-11T09:41:07.359 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-02-11T09:41:07.361 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:07.399 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:07 +0000 (0:00:00.064) 0:02:42.964 ***** 2021-02-11T09:41:07.425 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-11T09:41:07.426 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:07.459 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:07 +0000 (0:00:00.060) 0:02:43.024 ***** 2021-02-11T09:41:07.485 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-11T09:41:07.486 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:07.519 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:07 +0000 (0:00:00.060) 0:02:43.084 ***** 2021-02-11T09:41:07.658 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-02-11T09:41:07.660 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-02-11T09:41:07.694 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:07 +0000 (0:00:00.174) 0:02:43.259 ***** 2021-02-11T09:41:07.834 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-02-11T09:41:07.836 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:41:07.870 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:07 +0000 (0:00:00.176) 0:02:43.435 ***** 2021-02-11T09:41:07.877 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-02-11T09:41:07.878 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => {"changed": true} 2021-02-11T09:41:07.914 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:07 +0000 (0:00:00.044) 0:02:43.479 ***** 2021-02-11T09:41:08.135 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-02-11T09:41:08.137 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-02-11T09:41:08.179 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:08 +0000 (0:00:00.264) 0:02:43.744 ***** 2021-02-11T09:41:08.190 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-02-11T09:41:08.191 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-02-11T09:41:08.244 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:08 +0000 (0:00:00.065) 0:02:43.809 ***** 2021-02-11T09:41:08.251 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-02-11T09:41:08.251 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:08.294 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:08 +0000 (0:00:00.049) 0:02:43.859 ***** 2021-02-11T09:41:08.323 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-02-11T09:41:08.324 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:08.363 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:08 +0000 (0:00:00.069) 0:02:43.929 ***** 2021-02-11T09:41:08.390 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-11T09:41:08.392 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:08.427 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:08 +0000 (0:00:00.063) 0:02:43.992 ***** 2021-02-11T09:41:08.453 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-11T09:41:08.455 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:08.489 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:08 +0000 (0:00:00.061) 0:02:44.054 ***** 2021-02-11T09:41:08.565 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-02-11T09:41:08.566 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-02-11T09:41:08.600 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:08 +0000 (0:00:00.110) 0:02:44.165 ***** 2021-02-11T09:41:08.648 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-02-11T09:41:08.649 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-02-11T09:41:08.685 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:08 +0000 (0:00:00.084) 0:02:44.250 ***** 2021-02-11T09:41:08.815 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-02-11T09:41:08.816 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:41:08.851 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:08 +0000 (0:00:00.166) 0:02:44.417 ***** 2021-02-11T09:41:09.158 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-02-11T09:41:09.159 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-02-11T09:41:09.194 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:09 +0000 (0:00:00.342) 0:02:44.759 ***** 2021-02-11T09:41:09.504 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-02-11T09:41:09.506 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1613036134, "cache_updated": false, "changed": false} 2021-02-11T09:41:09.541 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:09 +0000 (0:00:00.346) 0:02:45.106 ***** 2021-02-11T09:41:09.548 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-02-11T09:41:09.548 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-02-11T09:41:09.585 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:09 +0000 (0:00:00.044) 0:02:45.150 ***** 2021-02-11T09:41:09.706 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-02-11T09:41:09.708 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.043705", "end": "2021-02-11 09:41:09.981081", "rc": 0, "start": "2021-02-11 09:41:09.937376", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-02-11T09:41:09.744 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:09 +0000 (0:00:00.158) 0:02:45.309 ***** 2021-02-11T09:41:09.771 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-02-11T09:41:09.772 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:09.806 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:09 +0000 (0:00:00.061) 0:02:45.371 ***** 2021-02-11T09:41:09.832 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-02-11T09:41:09.834 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:09.867 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:09 +0000 (0:00:00.061) 0:02:45.432 ***** 2021-02-11T09:41:10.506 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-02-11T09:41:10.508 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"cache_update_time": 1613036091, "cache_updated": false, "changed": false} 2021-02-11T09:41:10.540 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:10 +0000 (0:00:00.673) 0:02:46.105 ***** 2021-02-11T09:41:10.722 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-02-11T09:41:10.723 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.049253", "end": "2021-02-11 09:41:10.709355", "rc": 0, "start": "2021-02-11 09:41:10.660102", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-02-11T09:41:10.757 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:10 +0000 (0:00:00.217) 0:02:46.323 ***** 2021-02-11T09:41:10.763 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-02-11T09:41:10.763 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1613036134, "cache_updated": false, "changed": false} 2021-02-11T09:41:10.799 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:10 +0000 (0:00:00.041) 0:02:46.364 ***** 2021-02-11T09:41:10.804 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-02-11T09:41:10.805 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:10.839 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:10 +0000 (0:00:00.039) 0:02:46.404 ***** 2021-02-11T09:41:10.844 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-02-11T09:41:10.846 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:10.878 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:10 +0000 (0:00:00.039) 0:02:46.443 ***** 2021-02-11T09:41:10.884 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-02-11T09:41:10.885 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:10.919 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:10 +0000 (0:00:00.041) 0:02:46.484 ***** 2021-02-11T09:41:11.194 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-02-11T09:41:11.196 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-02-11T09:41:11.229 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:11 +0000 (0:00:00.310) 0:02:46.795 ***** 2021-02-11T09:41:11.255 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-02-11T09:41:11.257 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:11.290 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:11 +0000 (0:00:00.060) 0:02:46.855 ***** 2021-02-11T09:41:11.504 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-02-11T09:41:11.505 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 09:38:18 UTC", "ActiveEnterTimestampMonotonic": "27865788", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service sysinit.target network.target system.slice systemd-journald.socket -.mount basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 09:38:12 UTC", "AssertTimestampMonotonic": "21344127", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 09:38:12 UTC", "ConditionTimestampMonotonic": "21344127", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-02-11T09:41:11.506 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1833", "ExecMainStartTimestamp": "Thu 2021-02-11 09:38:18 UTC", "ExecMainStartTimestampMonotonic": "27865774", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 09:38:12 UTC", "InactiveExitTimestampMonotonic": "21345055", "InvocationID": "ce82eb5302c8466c8cf008b1959ad505", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-02-11T09:41:11.508 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1833", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 09:38:18 UTC", "S 2021-02-11T09:41:11.509 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "27865788", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 09:38:18 UTC", "WatchdogTimestampMonotonic": "27865786", "WatchdogUSec": "0"}} 2021-02-11T09:41:11.542 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:11 +0000 (0:00:00.251) 0:02:47.107 ***** 2021-02-11T09:41:11.578 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:41:11.579 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-02-11T09:41:11.613 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:11 +0000 (0:00:00.071) 0:02:47.178 ***** 2021-02-11T09:41:11.640 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:41:11.642 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:11.675 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:11 +0000 (0:00:00.062) 0:02:47.241 ***** 2021-02-11T09:41:11.702 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-02-11T09:41:11.704 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:11.737 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:11 +0000 (0:00:00.061) 0:02:47.302 ***** 2021-02-11T09:41:11.805 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-02-11T09:41:11.806 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"cache_update_time": 1613036091, "cache_updated": false, "changed": false} 2021-02-11T09:41:11.840 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:11 +0000 (0:00:00.103) 0:02:47.406 ***** 2021-02-11T09:41:11.867 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-02-11T09:41:11.868 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:11.901 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:11 +0000 (0:00:00.060) 0:02:47.466 ***** 2021-02-11T09:41:12.207 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-02-11T09:41:12.209 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-02-11T09:41:12.242 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:12 +0000 (0:00:00.340) 0:02:47.807 ***** 2021-02-11T09:41:12.268 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-02-11T09:41:12.270 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:12.301 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:12 +0000 (0:00:00.058) 0:02:47.866 ***** 2021-02-11T09:41:12.514 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-02-11T09:41:12.515 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 09:37:57 UTC", "ActiveEnterTimestampMonotonic": "24921289", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target -.mount sysinit.target systemd-journald.socket systemd-tmpfiles-setup.service system.slice basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 09:37:51 UTC", "AssertTimestampMonotonic": "18356181", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 09:37:51 UTC", "ConditionTimestampMonotonic": "18356180", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-02-11T09:41:12.516 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1747", "ExecMainStartTimestamp": "Thu 2021-02-11 09:37:57 UTC", "ExecMainStartTimestampMonotonic": "24921274", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 09:37:51 UTC", "InactiveExitTimestampMonotonic": "18357002", "InvocationID": "71ce9935d6d544b48f9e0194c0791f51", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-02-11T09:41:12.518 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1747", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 09:37:57 UTC", "S 2021-02-11T09:41:12.519 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "24921289", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 09:37:57 UTC", "WatchdogTimestampMonotonic": "24921287", "WatchdogUSec": "0"}} 2021-02-11T09:41:12.550 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:12 +0000 (0:00:00.248) 0:02:48.115 ***** 2021-02-11T09:41:12.576 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:41:12.578 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-02-11T09:41:12.610 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:12 +0000 (0:00:00.060) 0:02:48.175 ***** 2021-02-11T09:41:12.616 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-02-11T09:41:12.617 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1613036134, "cache_updated": false, "changed": false} 2021-02-11T09:41:12.651 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:12 +0000 (0:00:00.041) 0:02:48.216 ***** 2021-02-11T09:41:12.656 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:41:12.656 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:12.693 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:12 +0000 (0:00:00.042) 0:02:48.259 ***** 2021-02-11T09:41:12.699 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-02-11T09:41:12.700 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:12.733 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:12 +0000 (0:00:00.039) 0:02:48.298 ***** 2021-02-11T09:41:12.740 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-02-11T09:41:12.740 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:12.776 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:12 +0000 (0:00:00.042) 0:02:48.341 ***** 2021-02-11T09:41:12.865 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-02-11T09:41:12.867 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-02-11T09:41:12.900 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:12 +0000 (0:00:00.124) 0:02:48.466 ***** 2021-02-11T09:41:13.205 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-02-11T09:41:13.206 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-02-11T09:41:13.240 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:13 +0000 (0:00:00.339) 0:02:48.806 ***** 2021-02-11T09:41:13.701 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-02-11T09:41:13.702 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"cache_update_time": 1613036091, "cache_updated": false, "changed": false} 2021-02-11T09:41:13.736 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:13 +0000 (0:00:00.495) 0:02:49.301 ***** 2021-02-11T09:41:13.762 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-02-11T09:41:13.763 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:13.796 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:13 +0000 (0:00:00.059) 0:02:49.361 ***** 2021-02-11T09:41:13.916 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-02-11T09:41:13.917 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-02-11T09:41:13.950 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:13 +0000 (0:00:00.154) 0:02:49.516 ***** 2021-02-11T09:41:14.091 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-02-11T09:41:14.092 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-02-11T09:41:14.131 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:14 +0000 (0:00:00.180) 0:02:49.696 ***** 2021-02-11T09:41:14.262 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-02-11T09:41:14.263 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-02-11T09:41:14.298 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:14 +0000 (0:00:00.167) 0:02:49.863 ***** 2021-02-11T09:41:14.650 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-02-11T09:41:14.652 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false} 2021-02-11T09:41:14.685 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:14 +0000 (0:00:00.386) 0:02:50.250 ***** 2021-02-11T09:41:14.711 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-02-11T09:41:14.712 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-02-11T09:41:14.746 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:14 +0000 (0:00:00.060) 0:02:50.311 ***** 2021-02-11T09:41:14.888 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-02-11T09:41:14.889 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1613036078.036033, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1613036078.0320327, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 746, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1613036078.0320327, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3375002756", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-02-11T09:41:14.923 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:14 +0000 (0:00:00.177) 0:02:50.489 ***** 2021-02-11T09:41:15.105 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-02-11T09:41:19.179 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} Thursday 11 February 2021 09:41:15 +0000 (0:00:00.218) 0:02:50.707 ***** TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "msg": ""} Thursday 11 February 2021 09:41:15 +0000 (0:00:00.073) 0:02:50.780 ***** TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} Thursday 11 February 2021 09:41:15 +0000 (0:00:00.060) 0:02:50.841 ***** TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:15 +0000 (0:00:00.059) 0:02:50.901 ***** TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi133.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} Thursday 11 February 2021 09:4 2021-02-11T09:41:19.180 INFO:teuthology.task.ansible.out:1:15 +0000 (0:00:00.164) 0:02:51.066 ***** TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi133.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} Thursday 11 February 2021 09:41:15 +0000 (0:00:00.143) 0:02:51.209 ***** TASK [testnode : Purge resolvconf] ********************************************* ok: [smithi026.front.sepia.ceph.com] => {"changed": false} Thursday 11 February 2021 09:41:15 +0000 (0:00:00.050) 0:02:51.260 ***** TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:15 +0000 (0:00:00.045) 0:02:51.305 ***** TASK [testnode : Set interface] ************************************************ ok: [smithi026.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} Thursday 11 February 2021 09:41:15 +0000 (0:00:00.043) 0:02:51.348 ***** TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi133.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0" 2021-02-11T09:41:19.182 INFO:teuthology.task.ansible.out:, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:15 +0000 (0:00:00.056) 0:02:51.404 ***** TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi133.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional resu 2021-02-11T09:41:19.183 INFO:teuthology.task.ansible.out:lt was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:15 +0000 (0:00:00.097) 0:02:51.502 ***** TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1613036044.592952, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1613036044.5889518, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 746, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1613036044.5889518, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "3375002756", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} Thursday 11 February 2021 09:41:15 +0000 (0:00:00.046) 0:02:51.548 ***** TASK [testnode : set_fact] ***************************************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:16 +0000 (0:00:00.046) 0:02:51.595 ***** TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:16 +0000 (0:00:00.072) 0:02:51.668 ***** TASK [testnode : Rewrite /etc/network/interfaces to 2021-02-11T09:41:19.184 INFO:teuthology.task.ansible.out:use dhcp] ****************** ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "msg": ""} Thursday 11 February 2021 09:41:16 +0000 (0:00:00.052) 0:02:51.720 ***** TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:16 +0000 (0:00:00.049) 0:02:51.770 ***** TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi026.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} Thursday 11 February 2021 09:41:16 +0000 (0:00:00.042) 0:02:51.812 ***** TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:16 +0000 (0:00:00.042) 0:02:51.855 ***** TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:16 +0000 (0:00:00.042) 0:02:51.897 ***** TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:16 +0000 (0:00:00.047) 0:02:51.944 ***** TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:16 +0000 (0:00:00.077) 0:02:52.022 ***** TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi026.front.sepia.ceph.com] => {"backup": "", "chang 2021-02-11T09:41:19.186 INFO:teuthology.task.ansible.out:ed": true, "msg": "line replaced"} Thursday 11 February 2021 09:41:16 +0000 (0:00:00.052) 0:02:52.075 ***** TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:16 +0000 (0:00:00.044) 0:02:52.120 ***** TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:16 +0000 (0:00:00.061) 0:02:52.181 ***** TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi026.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} Thursday 11 February 2021 09:41:16 +0000 (0:00:00.052) 0:02:52.234 ***** TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:16 +0000 (0:00:00.059) 0:02:52.293 ***** TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi026.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi026.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} skipping: [smithi026.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi026.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-02-11T09:41:19.187 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi133.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011164", "end": "2021-02-11 09:41:17.033214", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-11 09:41:17.022050", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi026.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:16 +0000 (0:00:00.085) 0:02:52.379 ***** TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi026.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi026.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi026.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "chang 2021-02-11T09:41:19.188 INFO:teuthology.task.ansible.out:ed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi026.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi026.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:16 +0000 (0:00:00.084) 0:02:52.463 ***** TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi133.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006076", "end": "2021-02-11 09:41:17.158851", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-11 09:41:17.152775", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : set_fact] ***************************************************** skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:16 +0000 (0:00:00.051) 0:02:52.515 ***** TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditiona 2021-02-11T09:41:19.190 INFO:teuthology.task.ansible.out:l result was False"} Thursday 11 February 2021 09:41:17 +0000 (0:00:00.060) 0:02:52.576 ***** TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi133.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005440", "end": "2021-02-11 09:41:17.281779", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-11 09:41:17.276339", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:17 +0000 (0:00:00.063) 0:02:52.640 ***** TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:17 +0000 (0:00:00.063) 0:02:52.703 ***** TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:41:17 +0000 (0:00:00.063) 0:02:52.767 ***** TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi133.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.078315", "end": "2021-02-11 09:41:17.455620", "failed_when_result": false, "item": "/var/ 2021-02-11T09:41:19.190 INFO:teuthology.task.ansible.out:log/", "rc": 0, "start": "2021-02-11 09:41:17.377305", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:19.192 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-11T09:41:19.194 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:19.229 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:19 +0000 (0:00:02.027) 0:02:54.794 ***** 2021-02-11T09:41:19.264 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:19 +0000 (0:00:00.035) 0:02:54.829 ***** 2021-02-11T09:41:19.270 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-02-11T09:41:19.271 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:19.305 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:19 +0000 (0:00:00.040) 0:02:54.870 ***** 2021-02-11T09:41:19.310 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-02-11T09:41:19.343 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:19 +0000 (0:00:00.038) 0:02:54.908 ***** 2021-02-11T09:41:19.348 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-11T09:41:19.348 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:19.387 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:19 +0000 (0:00:00.043) 0:02:54.952 ***** 2021-02-11T09:41:19.478 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-02-11T09:41:19.479 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-02-11T09:41:19.567 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-11T09:41:19.569 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011123", "end": "2021-02-11 09:41:19.548051", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-11 09:41:19.536928", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:19.682 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006057", "end": "2021-02-11 09:41:19.667736", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-11 09:41:19.661679", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:19.796 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005640", "end": "2021-02-11 09:41:19.785255", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-11 09:41:19.779615", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:19.941 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.026725", "end": "2021-02-11 09:41:19.923609", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-02-11 09:41:19.896884", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:41:19.978 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:19 +0000 (0:00:00.591) 0:02:55.544 ***** 2021-02-11T09:41:20.005 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-02-11T09:41:20.042 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:20 +0000 (0:00:00.063) 0:02:55.607 ***** 2021-02-11T09:41:20.171 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-02-11T09:41:20.173 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-02-11T09:41:20.277 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-02-11T09:41:20.311 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:20 +0000 (0:00:00.269) 0:02:55.876 ***** 2021-02-11T09:41:20.347 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:20 +0000 (0:00:00.036) 0:02:55.912 ***** 2021-02-11T09:41:21.026 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi133.front.sepia.ceph.com] 2021-02-11T09:41:21.104 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:21 +0000 (0:00:00.757) 0:02:56.669 ***** 2021-02-11T09:41:21.146 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T09:41:21.147 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-02-11T09:41:21.183 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:21 +0000 (0:00:00.078) 0:02:56.748 ***** 2021-02-11T09:41:21.210 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T09:41:21.212 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:21.248 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:21 +0000 (0:00:00.064) 0:02:56.813 ***** 2021-02-11T09:41:21.284 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-02-11T09:41:21.286 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-02-11T09:41:21.320 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:21 +0000 (0:00:00.072) 0:02:56.885 ***** 2021-02-11T09:41:22.094 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi026.front.sepia.ceph.com] 2021-02-11T09:41:22.172 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:22 +0000 (0:00:00.852) 0:02:57.738 ***** 2021-02-11T09:41:22.210 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T09:41:22.212 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-02-11T09:41:22.250 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:22 +0000 (0:00:00.074) 0:02:57.812 ***** 2021-02-11T09:41:22.257 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-11T09:41:22.257 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1613036134, "cache_updated": false, "changed": false} 2021-02-11T09:41:22.294 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:22 +0000 (0:00:00.047) 0:02:57.859 ***** 2021-02-11T09:41:22.299 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T09:41:22.301 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:22.337 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:22 +0000 (0:00:00.043) 0:02:57.902 ***** 2021-02-11T09:41:22.343 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T09:41:22.343 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:22.384 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:22 +0000 (0:00:00.046) 0:02:57.949 ***** 2021-02-11T09:41:22.389 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-02-11T09:41:22.390 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-02-11T09:41:22.431 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:22 +0000 (0:00:00.047) 0:02:57.996 ***** 2021-02-11T09:41:22.441 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T09:41:22.441 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-02-11T09:41:22.480 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:22 +0000 (0:00:00.048) 0:02:58.045 ***** 2021-02-11T09:41:22.642 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-02-11T09:41:22.643 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:41:22.769 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:41:22.806 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:22 +0000 (0:00:00.326) 0:02:58.371 ***** 2021-02-11T09:41:23.132 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-02-11T09:41:23.134 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036482.840482-97424226778380/source", "state": "file", "uid": 0} 2021-02-11T09:41:23.381 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-11T09:41:23.383 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"cache_update_time": 1613036091, "cache_updated": false, "changed": false} 2021-02-11T09:41:23.419 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:23 +0000 (0:00:00.612) 0:02:58.984 ***** 2021-02-11T09:41:23.435 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-02-11T09:41:23.435 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036483.1391292-227117174407773/source", "state": "file", "uid": 0} 2021-02-11T09:41:23.472 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:23 +0000 (0:00:00.053) 0:02:59.037 ***** 2021-02-11T09:41:23.477 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T09:41:23.478 INFO:teuthology.task.ansible.out:skipping: [smithi026.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:41:23.515 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:23 +0000 (0:00:00.043) 0:02:59.080 ***** 2021-02-11T09:41:23.551 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T09:41:23.553 INFO:teuthology.task.ansible.out:ok: [smithi026.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-02-11T09:41:23.587 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:23 +0000 (0:00:00.071) 0:02:59.152 ***** 2021-02-11T09:41:23.747 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-02-11T09:41:23.748 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:41:23.862 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:41:23.898 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:23 +0000 (0:00:00.311) 0:02:59.463 ***** 2021-02-11T09:41:24.245 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-02-11T09:41:24.246 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036483.946039-238016798058334/source", "state": "file", "uid": 0} 2021-02-11T09:41:24.526 INFO:teuthology.task.ansible.out:changed: [smithi026.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036484.249474-192346227676046/source", "state": "file", "uid": 0} 2021-02-11T09:41:24.562 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:24 +0000 (0:00:00.663) 0:03:00.127 ***** 2021-02-11T09:41:26.910 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-02-11T09:41:26.911 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-knPr76-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-knPr76-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-02-11T09:41:26.912 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-02-11T09:41:26.947 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:26 +0000 (0:00:02.384) 0:03:02.512 ***** 2021-02-11T09:41:27.087 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-02-11T09:41:27.089 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-02-11T09:41:27.124 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:27 +0000 (0:00:00.177) 0:03:02.689 ***** 2021-02-11T09:41:27.428 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-02-11T09:41:27.429 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036487.1560357-34431681142053/source", "state": "file", "uid": 0} 2021-02-11T09:41:27.465 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:41:27 +0000 (0:00:00.341) 0:03:03.030 ***** 2021-02-11T09:41:27.792 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-02-11T09:41:43.253 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.181903", "end": "2021-02-11 09:41:28.049982", "rc": 0, "start": "2021-02-11 09:41:27.868079", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 09:41:27 +0000 (0:00:00.376) 0:03:03.407 ***** TASK [container-host : Install registries-conf-ctl] **************************** changed: [smithi026.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-jmvRkg-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running se 2021-02-11T09:41:43.254 INFO:teuthology.task.ansible.out:tup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-jmvRkg-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} Thursday 11 February 2021 09:41:28 +0000 (0:00:00.375) 0:03:03.783 ***** TASK [container-host : Check for docker's daemon.json] ************************* ok: [smithi026.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} Thursday 11 February 2021 09:41:28 +0000 (0:00:00.167) 2021-02-11T09:41:43.256 INFO:teuthology.task.ansible.out: 0:03:03.951 ***** TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ changed: [smithi026.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036488.416099-84836746948625/source", "state": "file", "uid": 0} Thursday 11 February 2021 09:41:28 +0000 (0:00:00.366) 0:03:04.317 ***** TASK [container-host : Add local docker.io registry mirror] ******************** changed: [smithi026.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.119792", "end": "2021-02-11 09:41:28.988588", "rc": 0, "start": "2021-02-11 09:41:28.868796", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 09:41:29 +0000 (0:00:00.282) 0:03:04.599 ***** TASK [container-host : Restart docker service] ********************************* changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "sysinit.target network-online.target basic.target systemd-journald.socket containerd.service firewalld.service system.slice docker.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate 2021-02-11T09:41:43.257 INFO:teuthology.task.ansible.out:": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709 2021-02-11T09:41:43.258 INFO:teuthology.task.ansible.out:551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": 2021-02-11T09:41:43.260 INFO:teuthology.task.ansible.out:"no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target docker.socket system.slice", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 09:40:21 UTC", "StateChangeTimestampMonotonic": "150767368", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "containerd.service network-online.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} changed: [smithi026.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", 2021-02-11T09:41:43.261 INFO:teuthology.task.ansible.out: "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice firewalld.service sysinit.target network-online.target containerd.service docker.socket basic.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start 2021-02-11T09:41:43.262 INFO:teuthology.task.ansible.out:_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEU 2021-02-11T09:41:43.262 INFO:teuthology.task.ansible.out:E": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": 2021-02-11T09:41:43.264 INFO:teuthology.task.ansible.out:"0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "docker.socket sysinit.target system.slice", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 09:40:21 UTC", "StateChangeTimestampMonotonic": "168439905", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", " 2021-02-11T09:41:43.265 INFO:teuthology.task.ansible.out:TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-online.target containerd.service", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-11T09:41:43.366 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-02-11T09:41:43.371 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-02-11T09:41:43.376 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* 2021-02-11T09:41:43.377 INFO:teuthology.task.ansible.out:smithi026.front.sepia.ceph.com : ok=126 changed=38 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 smithi133.front.sepia.ceph.com : ok=126 changed=38 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 Thursday 11 February 2021 09:41:43 +0000 (0:00:14.342) 0:03:18.941 ***** =============================================================================== users : Update authorized_keys using the keys repo --------------------- 46.93s users : Create all admin users with sudo access. ----------------------- 18.18s container-host : Restart docker service -------------------------------- 14.34s testnode : Zap all non-root disks -------------------------------------- 13.80s testnode : Ensure packages are not present. ----------------------------- 8.38s testnode : Install packages --------------------------------------------- 7.70s users : Remove revoked users -------------------------------------------- 6.37s common : Update apt cache ----------------------------------------------- 3.30s testnode : Update apt cache. -------------------------------------------- 3.07s container-host : Install registries-conf-ctl ---------------------------- 2.38s testnode : Bounce {{ nic_to_configure }} -------------------------------- 2.03s testnode : Install apt keys --------------------------------------------- 2.00s Gathering Facts --------------------------------------------------------- 1.89s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.85s users : Update authorized_keys for each user with literal keys ---------- 1.50s testnode : Create logical volume(s) ------------------------------------- 1.28s testnode : Install python-apt ------------------------------------------- 0.92s Gathering Facts --------------------------------------------------------- 0.83s testnode : Set the system's hostname ------------------------------------ 0.81s users : Check keys_repo HEAD sha1 --------------- 2021-02-11T09:41:43.378 INFO:teuthology.task.ansible.out:------------------------ 0.79s 2021-02-11T09:41:43.697 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi026.front.sepia.ceph.com'), Remote(name='ubuntu@smithi133.front.sepia.ceph.com')] 2021-02-11T09:41:43.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T09:41:43.700 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T09:41:43.777 DEBUG:teuthology.orchestra.run.smithi026:> true 2021-02-11T09:41:43.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T09:41:43.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T09:41:44.048 DEBUG:teuthology.orchestra.run.smithi133:> true 2021-02-11T09:41:44.235 INFO:teuthology.run_tasks:Running task clock... 2021-02-11T09:41:44.247 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-02-11T09:41:44.248 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-11T09:41:44.250 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-11T09:41:44.308 INFO:teuthology.orchestra.run.smithi026.stdout:11 Feb 09:41:44 ntpd[8310]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-02-11T09:41:44.308 INFO:teuthology.orchestra.run.smithi026.stdout:11 Feb 09:41:44 ntpd[8310]: Command line: ntpd -gq 2021-02-11T09:41:44.308 INFO:teuthology.orchestra.run.smithi026.stdout:11 Feb 09:41:44 ntpd[8310]: proto: precision = 0.126 usec (-23) 2021-02-11T09:41:44.310 INFO:teuthology.orchestra.run.smithi026.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-11T09:41:44.310 INFO:teuthology.orchestra.run.smithi026.stderr:restrict ::: KOD does nothing without LIMITED. 2021-02-11T09:41:44.310 INFO:teuthology.orchestra.run.smithi026.stdout:11 Feb 09:41:44 ntpd[8310]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-11T09:41:44.310 INFO:teuthology.orchestra.run.smithi026.stdout:11 Feb 09:41:44 ntpd[8310]: restrict ::: KOD does nothing without LIMITED. 2021-02-11T09:41:44.310 INFO:teuthology.orchestra.run.smithi026.stdout:11 Feb 09:41:44 ntpd[8310]: Listen and drop on 0 v6wildcard [::]:123 2021-02-11T09:41:44.310 INFO:teuthology.orchestra.run.smithi026.stdout:11 Feb 09:41:44 ntpd[8310]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-02-11T09:41:44.310 INFO:teuthology.orchestra.run.smithi026.stdout:11 Feb 09:41:44 ntpd[8310]: Listen normally on 2 lo 127.0.0.1:123 2021-02-11T09:41:44.310 INFO:teuthology.orchestra.run.smithi026.stdout:11 Feb 09:41:44 ntpd[8310]: Listen normally on 3 ens1f0 172.21.15.26:123 2021-02-11T09:41:44.310 INFO:teuthology.orchestra.run.smithi026.stdout:11 Feb 09:41:44 ntpd[8310]: Listen normally on 4 lo [::1]:123 2021-02-11T09:41:44.311 INFO:teuthology.orchestra.run.smithi026.stdout:11 Feb 09:41:44 ntpd[8310]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fea5:cac0%3]:123 2021-02-11T09:41:44.311 INFO:teuthology.orchestra.run.smithi026.stdout:11 Feb 09:41:44 ntpd[8310]: Listening on routing socket on fd #22 for interface updates 2021-02-11T09:41:45.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:45 ntpd[8310]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:45.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:45 ntpd[8310]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:45.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:45 ntpd[8310]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:45.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:45 ntpd[8310]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:47.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:47 ntpd[8310]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:47.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:47 ntpd[8310]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:47.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:47 ntpd[8310]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:47.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:47 ntpd[8310]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:48.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:48 ntpd[8310]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:48.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:48 ntpd[8310]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:48.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:48 ntpd[8310]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:48.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:48 ntpd[8310]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:48.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:48 ntpd[8310]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:48.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:48 ntpd[8310]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:48.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:48 ntpd[8310]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:48.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:48 ntpd[8310]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:48.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:48 ntpd[8310]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:48.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:48 ntpd[8310]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:48.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:48 ntpd[8310]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:48.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:48 ntpd[8310]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:49.308 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:49 ntpd[8310]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:49.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:49 ntpd[8310]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:49.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:49 ntpd[8310]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:49.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:49 ntpd[8310]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:50.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:50 ntpd[8310]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:50.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:50 ntpd[8310]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:50.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:50 ntpd[8310]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:50.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:50 ntpd[8310]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:50.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:50 ntpd[8310]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:50.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:50 ntpd[8310]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:50.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:50 ntpd[8310]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:50.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:50 ntpd[8310]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:50.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:50 ntpd[8310]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:50.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:50 ntpd[8310]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:50.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:50 ntpd[8310]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:50.310 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:50 ntpd[8310]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:51.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:51 ntpd[8310]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:51.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:51 ntpd[8310]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:51.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:51 ntpd[8310]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:51.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:51 ntpd[8310]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:51.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:51 ntpd[8310]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-02-11T09:41:51.309 INFO:teuthology.orchestra.run.smithi026.stderr:11 Feb 09:41:51 ntpd[8310]: can't open /var/log/ntpstats/loopstats.20210211: Permission denied 2021-02-11T09:41:51.309 INFO:teuthology.orchestra.run.smithi026.stdout:11 Feb 09:41:51 ntpd[8310]: ntpd: time slew -0.000482 s 2021-02-11T09:41:51.309 INFO:teuthology.orchestra.run.smithi026.stdout:ntpd: time slew -0.000482s 2021-02-11T09:41:51.359 INFO:teuthology.orchestra.run.smithi026.stdout: remote refid st t when poll reach delay offset jitter 2021-02-11T09:41:51.359 INFO:teuthology.orchestra.run.smithi026.stdout:============================================================================== 2021-02-11T09:41:51.359 INFO:teuthology.orchestra.run.smithi026.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T09:41:51.359 INFO:teuthology.orchestra.run.smithi026.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T09:41:51.359 INFO:teuthology.orchestra.run.smithi026.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T09:41:51.359 INFO:teuthology.orchestra.run.smithi026.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T09:41:51.360 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-11T09:41:51.360 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-11T09:41:51.413 INFO:teuthology.orchestra.run.smithi133.stdout:11 Feb 09:41:51 ntpd[8418]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-02-11T09:41:51.414 INFO:teuthology.orchestra.run.smithi133.stdout:11 Feb 09:41:51 ntpd[8418]: Command line: ntpd -gq 2021-02-11T09:41:51.414 INFO:teuthology.orchestra.run.smithi133.stdout:11 Feb 09:41:51 ntpd[8418]: proto: precision = 0.123 usec (-23) 2021-02-11T09:41:51.415 INFO:teuthology.orchestra.run.smithi133.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-11T09:41:51.415 INFO:teuthology.orchestra.run.smithi133.stderr:restrict ::: KOD does nothing without LIMITED. 2021-02-11T09:41:51.415 INFO:teuthology.orchestra.run.smithi133.stdout:11 Feb 09:41:51 ntpd[8418]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-11T09:41:51.416 INFO:teuthology.orchestra.run.smithi133.stdout:11 Feb 09:41:51 ntpd[8418]: restrict ::: KOD does nothing without LIMITED. 2021-02-11T09:41:51.416 INFO:teuthology.orchestra.run.smithi133.stdout:11 Feb 09:41:51 ntpd[8418]: Listen and drop on 0 v6wildcard [::]:123 2021-02-11T09:41:51.416 INFO:teuthology.orchestra.run.smithi133.stdout:11 Feb 09:41:51 ntpd[8418]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-02-11T09:41:51.416 INFO:teuthology.orchestra.run.smithi133.stdout:11 Feb 09:41:51 ntpd[8418]: Listen normally on 2 lo 127.0.0.1:123 2021-02-11T09:41:51.417 INFO:teuthology.orchestra.run.smithi133.stdout:11 Feb 09:41:51 ntpd[8418]: Listen normally on 3 enp3s0f1 172.21.15.133:123 2021-02-11T09:41:51.417 INFO:teuthology.orchestra.run.smithi133.stdout:11 Feb 09:41:51 ntpd[8418]: Listen normally on 4 lo [::1]:123 2021-02-11T09:41:51.417 INFO:teuthology.orchestra.run.smithi133.stdout:11 Feb 09:41:51 ntpd[8418]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8087%5]:123 2021-02-11T09:41:51.417 INFO:teuthology.orchestra.run.smithi133.stdout:11 Feb 09:41:51 ntpd[8418]: Listening on routing socket on fd #22 for interface updates 2021-02-11T09:41:52.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:52 ntpd[8418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:52.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:52 ntpd[8418]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:52.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:52 ntpd[8418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:52.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:52 ntpd[8418]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:52.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:52 ntpd[8418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:52.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:52 ntpd[8418]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:52.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:52 ntpd[8418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:52.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:52 ntpd[8418]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:54.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:54 ntpd[8418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:54.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:54 ntpd[8418]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:54.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:54 ntpd[8418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:54.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:54 ntpd[8418]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:54.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:54 ntpd[8418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:54.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:54 ntpd[8418]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:54.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:54 ntpd[8418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:54.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:54 ntpd[8418]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:55.414 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:55 ntpd[8418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:55.414 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:55 ntpd[8418]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:55.414 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:55 ntpd[8418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:55.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:55 ntpd[8418]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:56.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:56 ntpd[8418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:56.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:56 ntpd[8418]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:56.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:56 ntpd[8418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:56.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:56 ntpd[8418]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:56.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:56 ntpd[8418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:56.416 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:56 ntpd[8418]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:56.416 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:56 ntpd[8418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:56.416 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:56 ntpd[8418]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:56.416 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:56 ntpd[8418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:56.416 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:56 ntpd[8418]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:56.416 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:56 ntpd[8418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:56.416 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:56 ntpd[8418]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:57.414 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:57 ntpd[8418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:41:57.414 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:57 ntpd[8418]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:41:57.414 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:57 ntpd[8418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:41:57.415 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:57 ntpd[8418]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:41:58.414 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:58 ntpd[8418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:42:22.893 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:58 ntpd[8418]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:42:22.893 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:58 ntpd[8418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:42:22.894 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:58 ntpd[8418]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:42:22.894 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:58 ntpd[8418]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-02-11T09:42:22.894 INFO:teuthology.orchestra.run.smithi133.stderr:11 Feb 09:41:58 ntpd[8418]: can't open /var/log/ntpstats/loopstats.20210211: Permission denied 2021-02-11T09:42:22.894 INFO:teuthology.orchestra.run.smithi133.stdout:11 Feb 09:41:58 ntpd[8418]: ntpd: time slew -0.000375 s 2021-02-11T09:42:22.894 INFO:teuthology.orchestra.run.smithi133.stdout:ntpd: time slew -0.000375s 2021-02-11T09:42:22.896 INFO:teuthology.orchestra.run.smithi133.stdout: remote refid st t when poll reach delay offset jitter 2021-02-11T09:42:22.896 INFO:teuthology.orchestra.run.smithi133.stdout:============================================================================== 2021-02-11T09:42:22.896 INFO:teuthology.orchestra.run.smithi133.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T09:42:22.896 INFO:teuthology.orchestra.run.smithi133.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T09:42:22.896 INFO:teuthology.orchestra.run.smithi133.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T09:42:22.896 INFO:teuthology.orchestra.run.smithi133.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T09:42:22.897 INFO:teuthology.run_tasks:Running task install... 2021-02-11T09:42:22.905 DEBUG:teuthology.task.install:project ceph 2021-02-11T09:42:22.906 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f'}} 2021-02-11T09:42:22.906 DEBUG:teuthology.task.install:config {'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f'} 2021-02-11T09:42:22.906 INFO:teuthology.task.install:Using flavor: basic 2021-02-11T09:42:22.921 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-02-11T09:42:22.921 INFO:teuthology.task.install:extra packages: [] 2021-02-11T09:42:22.921 DEBUG:teuthology.orchestra.run.smithi026:> sudo lsb_release -is 2021-02-11T09:42:22.976 INFO:teuthology.orchestra.run.smithi026.stdout:Ubuntu 2021-02-11T09:42:22.982 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T09:42:22.982 DEBUG:teuthology.orchestra.run.smithi133:> sudo lsb_release -is 2021-02-11T09:42:22.982 DEBUG:teuthology.orchestra.run.smithi026:> sudo apt-key list | grep Ceph 2021-02-11T09:42:23.036 INFO:teuthology.orchestra.run.smithi133.stdout:Ubuntu 2021-02-11T09:42:23.041 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T09:42:23.041 DEBUG:teuthology.orchestra.run.smithi133:> sudo apt-key list | grep Ceph 2021-02-11T09:42:23.072 INFO:teuthology.orchestra.run.smithi026.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-02-11T09:42:23.135 INFO:teuthology.orchestra.run.smithi026.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-02-11T09:42:23.135 INFO:teuthology.orchestra.run.smithi026.stdout:uid [ unknown] Ceph.com (release key) 2021-02-11T09:42:23.136 DEBUG:teuthology.orchestra.run.smithi026:> sudo lsb_release -is 2021-02-11T09:42:23.137 INFO:teuthology.orchestra.run.smithi133.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-02-11T09:42:23.187 INFO:teuthology.orchestra.run.smithi026.stdout:Ubuntu 2021-02-11T09:42:23.191 INFO:teuthology.orchestra.run.smithi133.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-02-11T09:42:23.191 INFO:teuthology.orchestra.run.smithi133.stdout:uid [ unknown] Ceph.com (release key) 2021-02-11T09:42:23.192 DEBUG:teuthology.orchestra.run.smithi133:> sudo lsb_release -is 2021-02-11T09:42:23.193 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T09:42:23.193 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-02-11T09:42:23.193 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=5e5bb33c614118d912ca83c2e5f2be67b3acc04f 2021-02-11T09:42:23.246 INFO:teuthology.orchestra.run.smithi133.stdout:Ubuntu 2021-02-11T09:42:23.251 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T09:42:23.251 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-02-11T09:42:23.251 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=5e5bb33c614118d912ca83c2e5f2be67b3acc04f 2021-02-11T09:42:23.378 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default/ 2021-02-11T09:42:23.378 INFO:teuthology.task.install.deb:Package version is 17.0.0-735-g5e5bb33c-1bionic 2021-02-11T09:42:23.416 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default/ 2021-02-11T09:42:23.417 INFO:teuthology.task.install.deb:Package version is 17.0.0-735-g5e5bb33c-1bionic 2021-02-11T09:42:23.503 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:42:23.504 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-02-11T09:42:23.519 DEBUG:teuthology.orchestra.run.smithi026:> sudo apt-get update 2021-02-11T09:42:23.538 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:42:23.538 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-02-11T09:42:23.554 DEBUG:teuthology.orchestra.run.smithi133:> sudo apt-get update 2021-02-11T09:42:23.709 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-02-11T09:42:23.768 INFO:teuthology.orchestra.run.smithi026.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic InRelease 2021-02-11T09:42:23.791 INFO:teuthology.orchestra.run.smithi026.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-02-11T09:42:23.800 INFO:teuthology.orchestra.run.smithi133.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic InRelease 2021-02-11T09:42:23.816 INFO:teuthology.orchestra.run.smithi026.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic Release.gpg 2021-02-11T09:42:23.823 INFO:teuthology.orchestra.run.smithi133.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-02-11T09:42:23.833 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T09:42:23.843 INFO:teuthology.orchestra.run.smithi026.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,096 B] 2021-02-11T09:42:23.858 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-02-11T09:42:23.858 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T09:42:23.859 INFO:teuthology.orchestra.run.smithi133.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic Release.gpg 2021-02-11T09:42:23.884 INFO:teuthology.orchestra.run.smithi133.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,096 B] 2021-02-11T09:42:23.884 INFO:teuthology.orchestra.run.smithi026.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.6 kB] 2021-02-11T09:42:23.911 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-02-11T09:42:23.928 INFO:teuthology.orchestra.run.smithi133.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.6 kB] 2021-02-11T09:42:23.929 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-02-11T09:42:24.008 INFO:teuthology.orchestra.run.smithi026.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-02-11T09:42:24.010 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-02-11T09:42:24.316 INFO:teuthology.orchestra.run.smithi026.stdout:Fetched 27.3 kB in 1s (46.1 kB/s) 2021-02-11T09:42:24.464 INFO:teuthology.orchestra.run.smithi133.stdout:Fetched 27.3 kB in 1s (38.6 kB/s) 2021-02-11T09:42:25.609 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2021-02-11T09:42:25.656 DEBUG:teuthology.orchestra.run.smithi026:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-735-g5e5bb33c-1bionic cephadm=17.0.0-735-g5e5bb33c-1bionic ceph-mds=17.0.0-735-g5e5bb33c-1bionic ceph-mgr=17.0.0-735-g5e5bb33c-1bionic ceph-common=17.0.0-735-g5e5bb33c-1bionic ceph-fuse=17.0.0-735-g5e5bb33c-1bionic ceph-test=17.0.0-735-g5e5bb33c-1bionic radosgw=17.0.0-735-g5e5bb33c-1bionic python3-rados=17.0.0-735-g5e5bb33c-1bionic python3-rgw=17.0.0-735-g5e5bb33c-1bionic python3-cephfs=17.0.0-735-g5e5bb33c-1bionic python3-rbd=17.0.0-735-g5e5bb33c-1bionic libcephfs2=17.0.0-735-g5e5bb33c-1bionic libcephfs-dev=17.0.0-735-g5e5bb33c-1bionic librados2=17.0.0-735-g5e5bb33c-1bionic librbd1=17.0.0-735-g5e5bb33c-1bionic rbd-fuse=17.0.0-735-g5e5bb33c-1bionic 2021-02-11T09:42:25.705 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2021-02-11T09:42:25.740 INFO:teuthology.orchestra.run.smithi026.stdout:Reading package lists... 2021-02-11T09:42:25.753 DEBUG:teuthology.orchestra.run.smithi133:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-735-g5e5bb33c-1bionic cephadm=17.0.0-735-g5e5bb33c-1bionic ceph-mds=17.0.0-735-g5e5bb33c-1bionic ceph-mgr=17.0.0-735-g5e5bb33c-1bionic ceph-common=17.0.0-735-g5e5bb33c-1bionic ceph-fuse=17.0.0-735-g5e5bb33c-1bionic ceph-test=17.0.0-735-g5e5bb33c-1bionic radosgw=17.0.0-735-g5e5bb33c-1bionic python3-rados=17.0.0-735-g5e5bb33c-1bionic python3-rgw=17.0.0-735-g5e5bb33c-1bionic python3-cephfs=17.0.0-735-g5e5bb33c-1bionic python3-rbd=17.0.0-735-g5e5bb33c-1bionic libcephfs2=17.0.0-735-g5e5bb33c-1bionic libcephfs-dev=17.0.0-735-g5e5bb33c-1bionic librados2=17.0.0-735-g5e5bb33c-1bionic librbd1=17.0.0-735-g5e5bb33c-1bionic rbd-fuse=17.0.0-735-g5e5bb33c-1bionic 2021-02-11T09:42:25.830 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2021-02-11T09:42:26.008 INFO:teuthology.orchestra.run.smithi026.stdout:Building dependency tree... 2021-02-11T09:42:26.008 INFO:teuthology.orchestra.run.smithi026.stdout:Reading state information... 2021-02-11T09:42:26.089 INFO:teuthology.orchestra.run.smithi133.stdout:Building dependency tree... 2021-02-11T09:42:26.089 INFO:teuthology.orchestra.run.smithi133.stdout:Reading state information... 2021-02-11T09:42:26.187 INFO:teuthology.orchestra.run.smithi026.stdout:The following additional packages will be installed: 2021-02-11T09:42:26.187 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-02-11T09:42:26.187 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-02-11T09:42:26.188 INFO:teuthology.orchestra.run.smithi026.stdout: libjaeger libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-02-11T09:42:26.188 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-02-11T09:42:26.188 INFO:teuthology.orchestra.run.smithi026.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-02-11T09:42:26.189 INFO:teuthology.orchestra.run.smithi026.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-11T09:42:26.189 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-02-11T09:42:26.189 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-02-11T09:42:26.189 INFO:teuthology.orchestra.run.smithi026.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-02-11T09:42:26.189 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-02-11T09:42:26.189 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-02-11T09:42:26.189 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-02-11T09:42:26.189 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-02-11T09:42:26.189 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-02-11T09:42:26.189 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-02-11T09:42:26.189 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-02-11T09:42:26.189 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-02-11T09:42:26.190 INFO:teuthology.orchestra.run.smithi026.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-02-11T09:42:26.190 INFO:teuthology.orchestra.run.smithi026.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-02-11T09:42:26.190 INFO:teuthology.orchestra.run.smithi026.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-02-11T09:42:26.190 INFO:teuthology.orchestra.run.smithi026.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-02-11T09:42:26.190 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-02-11T09:42:26.190 INFO:teuthology.orchestra.run.smithi026.stdout:Suggested packages: 2021-02-11T09:42:26.190 INFO:teuthology.orchestra.run.smithi026.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-02-11T09:42:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-02-11T09:42:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-02-11T09:42:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-02-11T09:42:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-02-11T09:42:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-02-11T09:42:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-02-11T09:42:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-02-11T09:42:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-02-11T09:42:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-02-11T09:42:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-02-11T09:42:26.191 INFO:teuthology.orchestra.run.smithi026.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-02-11T09:42:26.192 INFO:teuthology.orchestra.run.smithi026.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-02-11T09:42:26.192 INFO:teuthology.orchestra.run.smithi026.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-02-11T09:42:26.192 INFO:teuthology.orchestra.run.smithi026.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-02-11T09:42:26.262 INFO:teuthology.orchestra.run.smithi133.stdout:The following additional packages will be installed: 2021-02-11T09:42:26.262 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-02-11T09:42:26.264 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-02-11T09:42:26.264 INFO:teuthology.orchestra.run.smithi133.stdout: libjaeger libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-02-11T09:42:26.264 INFO:teuthology.orchestra.run.smithi133.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-02-11T09:42:26.264 INFO:teuthology.orchestra.run.smithi133.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-02-11T09:42:26.264 INFO:teuthology.orchestra.run.smithi133.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-11T09:42:26.264 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-02-11T09:42:26.264 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-02-11T09:42:26.264 INFO:teuthology.orchestra.run.smithi133.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-02-11T09:42:26.265 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-02-11T09:42:26.265 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-02-11T09:42:26.265 INFO:teuthology.orchestra.run.smithi133.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-02-11T09:42:26.265 INFO:teuthology.orchestra.run.smithi133.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-02-11T09:42:26.265 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-02-11T09:42:26.265 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-02-11T09:42:26.265 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-02-11T09:42:26.265 INFO:teuthology.orchestra.run.smithi133.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-02-11T09:42:26.265 INFO:teuthology.orchestra.run.smithi133.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-02-11T09:42:26.265 INFO:teuthology.orchestra.run.smithi133.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-02-11T09:42:26.265 INFO:teuthology.orchestra.run.smithi133.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-02-11T09:42:26.265 INFO:teuthology.orchestra.run.smithi133.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-02-11T09:42:26.266 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-02-11T09:42:26.266 INFO:teuthology.orchestra.run.smithi133.stdout:Suggested packages: 2021-02-11T09:42:26.266 INFO:teuthology.orchestra.run.smithi133.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-02-11T09:42:26.266 INFO:teuthology.orchestra.run.smithi133.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-02-11T09:42:26.266 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-02-11T09:42:26.266 INFO:teuthology.orchestra.run.smithi133.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-02-11T09:42:26.266 INFO:teuthology.orchestra.run.smithi133.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-02-11T09:42:26.266 INFO:teuthology.orchestra.run.smithi133.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-02-11T09:42:26.266 INFO:teuthology.orchestra.run.smithi133.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-02-11T09:42:26.267 INFO:teuthology.orchestra.run.smithi133.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-02-11T09:42:26.267 INFO:teuthology.orchestra.run.smithi133.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-02-11T09:42:26.267 INFO:teuthology.orchestra.run.smithi133.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-02-11T09:42:26.267 INFO:teuthology.orchestra.run.smithi133.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-02-11T09:42:26.267 INFO:teuthology.orchestra.run.smithi133.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-02-11T09:42:26.267 INFO:teuthology.orchestra.run.smithi133.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-02-11T09:42:26.267 INFO:teuthology.orchestra.run.smithi133.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-02-11T09:42:26.267 INFO:teuthology.orchestra.run.smithi133.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-02-11T09:42:26.298 INFO:teuthology.orchestra.run.smithi026.stdout:The following NEW packages will be installed: 2021-02-11T09:42:26.299 INFO:teuthology.orchestra.run.smithi026.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-02-11T09:42:26.299 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-02-11T09:42:26.299 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-02-11T09:42:26.299 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libleveldb1v5 liblttng-ust0 2021-02-11T09:42:26.300 INFO:teuthology.orchestra.run.smithi026.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-02-11T09:42:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-02-11T09:42:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-02-11T09:42:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-11T09:42:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-02-11T09:42:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-02-11T09:42:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-02-11T09:42:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-02-11T09:42:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-02-11T09:42:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-02-11T09:42:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-02-11T09:42:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-02-11T09:42:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-02-11T09:42:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-02-11T09:42:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-02-11T09:42:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-02-11T09:42:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-02-11T09:42:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-02-11T09:42:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-02-11T09:42:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-02-11T09:42:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-02-11T09:42:26.303 INFO:teuthology.orchestra.run.smithi026.stdout:The following packages will be upgraded: 2021-02-11T09:42:26.303 INFO:teuthology.orchestra.run.smithi026.stdout: librados2 librbd1 2021-02-11T09:42:26.365 INFO:teuthology.orchestra.run.smithi133.stdout:The following NEW packages will be installed: 2021-02-11T09:42:26.365 INFO:teuthology.orchestra.run.smithi133.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-02-11T09:42:26.365 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-02-11T09:42:26.365 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-02-11T09:42:26.366 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libleveldb1v5 liblttng-ust0 2021-02-11T09:42:26.366 INFO:teuthology.orchestra.run.smithi133.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-02-11T09:42:26.366 INFO:teuthology.orchestra.run.smithi133.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-02-11T09:42:26.366 INFO:teuthology.orchestra.run.smithi133.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-02-11T09:42:26.367 INFO:teuthology.orchestra.run.smithi133.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-11T09:42:26.367 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-02-11T09:42:26.367 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-02-11T09:42:26.367 INFO:teuthology.orchestra.run.smithi133.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-02-11T09:42:26.367 INFO:teuthology.orchestra.run.smithi133.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-02-11T09:42:26.367 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-02-11T09:42:26.367 INFO:teuthology.orchestra.run.smithi133.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-02-11T09:42:26.367 INFO:teuthology.orchestra.run.smithi133.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-02-11T09:42:26.367 INFO:teuthology.orchestra.run.smithi133.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-02-11T09:42:26.367 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-02-11T09:42:26.367 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-02-11T09:42:26.367 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-02-11T09:42:26.368 INFO:teuthology.orchestra.run.smithi133.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-02-11T09:42:26.368 INFO:teuthology.orchestra.run.smithi133.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-02-11T09:42:26.368 INFO:teuthology.orchestra.run.smithi133.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-02-11T09:42:26.368 INFO:teuthology.orchestra.run.smithi133.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-02-11T09:42:26.368 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-02-11T09:42:26.368 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-02-11T09:42:26.368 INFO:teuthology.orchestra.run.smithi133.stdout:The following packages will be upgraded: 2021-02-11T09:42:26.369 INFO:teuthology.orchestra.run.smithi133.stdout: librados2 librbd1 2021-02-11T09:42:26.443 INFO:teuthology.orchestra.run.smithi026.stdout:2 upgraded, 111 newly installed, 0 to remove and 19 not upgraded. 2021-02-11T09:42:26.444 INFO:teuthology.orchestra.run.smithi026.stdout:Need to get 167 MB of archives. 2021-02-11T09:42:26.444 INFO:teuthology.orchestra.run.smithi026.stdout:After this operation, 778 MB of additional disk space will be used. 2021-02-11T09:42:26.444 INFO:teuthology.orchestra.run.smithi026.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 libjaeger amd64 17.0.0-735-g5e5bb33c-1bionic [3,720 B] 2021-02-11T09:42:26.489 INFO:teuthology.orchestra.run.smithi026.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-02-11T09:42:26.489 INFO:teuthology.orchestra.run.smithi026.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 17.0.0-735-g5e5bb33c-1bionic [3,123 kB] 2021-02-11T09:42:26.506 INFO:teuthology.orchestra.run.smithi133.stdout:2 upgraded, 111 newly installed, 0 to remove and 19 not upgraded. 2021-02-11T09:42:26.506 INFO:teuthology.orchestra.run.smithi133.stdout:Need to get 167 MB of archives. 2021-02-11T09:42:26.506 INFO:teuthology.orchestra.run.smithi133.stdout:After this operation, 778 MB of additional disk space will be used. 2021-02-11T09:42:26.506 INFO:teuthology.orchestra.run.smithi133.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 libjaeger amd64 17.0.0-735-g5e5bb33c-1bionic [3,720 B] 2021-02-11T09:42:26.553 INFO:teuthology.orchestra.run.smithi133.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 17.0.0-735-g5e5bb33c-1bionic [3,123 kB] 2021-02-11T09:42:26.555 INFO:teuthology.orchestra.run.smithi133.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-02-11T09:42:26.567 INFO:teuthology.orchestra.run.smithi026.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-02-11T09:42:26.635 INFO:teuthology.orchestra.run.smithi133.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-02-11T09:42:26.691 INFO:teuthology.orchestra.run.smithi026.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 17.0.0-735-g5e5bb33c-1bionic [3,162 kB] 2021-02-11T09:42:26.807 INFO:teuthology.orchestra.run.smithi026.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-02-11T09:42:26.808 INFO:teuthology.orchestra.run.smithi026.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-02-11T09:42:26.818 INFO:teuthology.orchestra.run.smithi026.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-02-11T09:42:26.859 INFO:teuthology.orchestra.run.smithi026.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 17.0.0-735-g5e5bb33c-1bionic [665 kB] 2021-02-11T09:42:26.883 INFO:teuthology.orchestra.run.smithi026.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 17.0.0-735-g5e5bb33c-1bionic [340 kB] 2021-02-11T09:42:26.883 INFO:teuthology.orchestra.run.smithi133.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-02-11T09:42:26.891 INFO:teuthology.orchestra.run.smithi133.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-02-11T09:42:26.895 INFO:teuthology.orchestra.run.smithi133.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-02-11T09:42:26.902 INFO:teuthology.orchestra.run.smithi133.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 17.0.0-735-g5e5bb33c-1bionic [3,162 kB] 2021-02-11T09:42:26.910 INFO:teuthology.orchestra.run.smithi026.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-02-11T09:42:26.910 INFO:teuthology.orchestra.run.smithi026.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 17.0.0-735-g5e5bb33c-1bionic [21.7 kB] 2021-02-11T09:42:26.912 INFO:teuthology.orchestra.run.smithi026.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 17.0.0-735-g5e5bb33c-1bionic [163 kB] 2021-02-11T09:42:26.913 INFO:teuthology.orchestra.run.smithi026.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-02-11T09:42:26.916 INFO:teuthology.orchestra.run.smithi026.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 17.0.0-735-g5e5bb33c-1bionic [29.2 kB] 2021-02-11T09:42:26.917 INFO:teuthology.orchestra.run.smithi026.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 17.0.0-735-g5e5bb33c-1bionic [335 kB] 2021-02-11T09:42:26.928 INFO:teuthology.orchestra.run.smithi026.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 17.0.0-735-g5e5bb33c-1bionic [3,350 kB] 2021-02-11T09:42:26.970 INFO:teuthology.orchestra.run.smithi026.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-02-11T09:42:26.975 INFO:teuthology.orchestra.run.smithi026.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-02-11T09:42:26.982 INFO:teuthology.orchestra.run.smithi026.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-02-11T09:42:26.989 INFO:teuthology.orchestra.run.smithi026.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-02-11T09:42:26.995 INFO:teuthology.orchestra.run.smithi026.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-02-11T09:42:27.006 INFO:teuthology.orchestra.run.smithi133.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-02-11T09:42:27.009 INFO:teuthology.orchestra.run.smithi133.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-02-11T09:42:27.013 INFO:teuthology.orchestra.run.smithi026.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-02-11T09:42:27.029 INFO:teuthology.orchestra.run.smithi026.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-02-11T09:42:27.044 INFO:teuthology.orchestra.run.smithi026.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-02-11T09:42:27.051 INFO:teuthology.orchestra.run.smithi133.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-02-11T09:42:27.057 INFO:teuthology.orchestra.run.smithi026.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-02-11T09:42:27.058 INFO:teuthology.orchestra.run.smithi133.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-02-11T09:42:27.067 INFO:teuthology.orchestra.run.smithi133.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-02-11T09:42:27.078 INFO:teuthology.orchestra.run.smithi133.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-02-11T09:42:27.088 INFO:teuthology.orchestra.run.smithi133.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-02-11T09:42:27.098 INFO:teuthology.orchestra.run.smithi026.stdout:Get:27 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 17.0.0-735-g5e5bb33c-1bionic [99.4 kB] 2021-02-11T09:42:27.102 INFO:teuthology.orchestra.run.smithi026.stdout:Get:28 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 17.0.0-735-g5e5bb33c-1bionic [414 kB] 2021-02-11T09:42:27.111 INFO:teuthology.orchestra.run.smithi133.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-02-11T09:42:27.113 INFO:teuthology.orchestra.run.smithi133.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-02-11T09:42:27.115 INFO:teuthology.orchestra.run.smithi026.stdout:Get:29 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 17.0.0-735-g5e5bb33c-1bionic [21.2 MB] 2021-02-11T09:42:27.123 INFO:teuthology.orchestra.run.smithi026.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-02-11T09:42:27.123 INFO:teuthology.orchestra.run.smithi133.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-02-11T09:42:27.124 INFO:teuthology.orchestra.run.smithi026.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-02-11T09:42:27.124 INFO:teuthology.orchestra.run.smithi026.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-02-11T09:42:27.127 INFO:teuthology.orchestra.run.smithi026.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-02-11T09:42:27.127 INFO:teuthology.orchestra.run.smithi026.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-02-11T09:42:27.128 INFO:teuthology.orchestra.run.smithi026.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-02-11T09:42:27.130 INFO:teuthology.orchestra.run.smithi026.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-02-11T09:42:27.133 INFO:teuthology.orchestra.run.smithi026.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-02-11T09:42:27.134 INFO:teuthology.orchestra.run.smithi133.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-02-11T09:42:27.201 INFO:teuthology.orchestra.run.smithi026.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-02-11T09:42:27.201 INFO:teuthology.orchestra.run.smithi026.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-02-11T09:42:27.202 INFO:teuthology.orchestra.run.smithi133.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-02-11T09:42:27.203 INFO:teuthology.orchestra.run.smithi133.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-02-11T09:42:27.203 INFO:teuthology.orchestra.run.smithi133.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-02-11T09:42:27.206 INFO:teuthology.orchestra.run.smithi133.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-02-11T09:42:27.207 INFO:teuthology.orchestra.run.smithi133.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-02-11T09:42:27.207 INFO:teuthology.orchestra.run.smithi133.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-02-11T09:42:27.210 INFO:teuthology.orchestra.run.smithi133.stdout:Get:26 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 17.0.0-735-g5e5bb33c-1bionic [665 kB] 2021-02-11T09:42:27.210 INFO:teuthology.orchestra.run.smithi133.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-02-11T09:42:27.212 INFO:teuthology.orchestra.run.smithi133.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-02-11T09:42:27.279 INFO:teuthology.orchestra.run.smithi026.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-02-11T09:42:27.280 INFO:teuthology.orchestra.run.smithi133.stdout:Get:29 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 17.0.0-735-g5e5bb33c-1bionic [340 kB] 2021-02-11T09:42:27.280 INFO:teuthology.orchestra.run.smithi026.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-02-11T09:42:27.280 INFO:teuthology.orchestra.run.smithi026.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-02-11T09:42:27.281 INFO:teuthology.orchestra.run.smithi133.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-02-11T09:42:27.282 INFO:teuthology.orchestra.run.smithi026.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-02-11T09:42:27.282 INFO:teuthology.orchestra.run.smithi133.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-02-11T09:42:27.284 INFO:teuthology.orchestra.run.smithi026.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-02-11T09:42:27.292 INFO:teuthology.orchestra.run.smithi026.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-02-11T09:42:27.296 INFO:teuthology.orchestra.run.smithi026.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-02-11T09:42:27.296 INFO:teuthology.orchestra.run.smithi026.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-02-11T09:42:27.332 INFO:teuthology.orchestra.run.smithi133.stdout:Get:32 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 17.0.0-735-g5e5bb33c-1bionic [21.7 kB] 2021-02-11T09:42:27.340 INFO:teuthology.orchestra.run.smithi133.stdout:Get:33 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 17.0.0-735-g5e5bb33c-1bionic [163 kB] 2021-02-11T09:42:27.345 INFO:teuthology.orchestra.run.smithi133.stdout:Get:34 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 17.0.0-735-g5e5bb33c-1bionic [29.2 kB] 2021-02-11T09:42:27.347 INFO:teuthology.orchestra.run.smithi133.stdout:Get:35 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 17.0.0-735-g5e5bb33c-1bionic [335 kB] 2021-02-11T09:42:27.357 INFO:teuthology.orchestra.run.smithi026.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-02-11T09:42:27.357 INFO:teuthology.orchestra.run.smithi133.stdout:Get:36 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 17.0.0-735-g5e5bb33c-1bionic [3,350 kB] 2021-02-11T09:42:27.360 INFO:teuthology.orchestra.run.smithi133.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-02-11T09:42:27.361 INFO:teuthology.orchestra.run.smithi133.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-02-11T09:42:27.361 INFO:teuthology.orchestra.run.smithi133.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-02-11T09:42:27.362 INFO:teuthology.orchestra.run.smithi133.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-02-11T09:42:27.366 INFO:teuthology.orchestra.run.smithi133.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-02-11T09:42:27.373 INFO:teuthology.orchestra.run.smithi133.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-02-11T09:42:27.378 INFO:teuthology.orchestra.run.smithi133.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-02-11T09:42:27.378 INFO:teuthology.orchestra.run.smithi133.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-02-11T09:42:27.440 INFO:teuthology.orchestra.run.smithi133.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-02-11T09:42:27.441 INFO:teuthology.orchestra.run.smithi026.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-02-11T09:42:27.441 INFO:teuthology.orchestra.run.smithi026.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-02-11T09:42:27.519 INFO:teuthology.orchestra.run.smithi133.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-02-11T09:42:27.519 INFO:teuthology.orchestra.run.smithi133.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-02-11T09:42:27.682 INFO:teuthology.orchestra.run.smithi133.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 17.0.0-735-g5e5bb33c-1bionic [99.4 kB] 2021-02-11T09:42:27.712 INFO:teuthology.orchestra.run.smithi133.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 17.0.0-735-g5e5bb33c-1bionic [414 kB] 2021-02-11T09:42:27.719 INFO:teuthology.orchestra.run.smithi026.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-02-11T09:42:27.726 INFO:teuthology.orchestra.run.smithi133.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 17.0.0-735-g5e5bb33c-1bionic [21.2 MB] 2021-02-11T09:42:27.759 INFO:teuthology.orchestra.run.smithi026.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-02-11T09:42:27.760 INFO:teuthology.orchestra.run.smithi026.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-02-11T09:42:27.801 INFO:teuthology.orchestra.run.smithi133.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-02-11T09:42:27.803 INFO:teuthology.orchestra.run.smithi026.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-02-11T09:42:27.803 INFO:teuthology.orchestra.run.smithi026.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-02-11T09:42:27.804 INFO:teuthology.orchestra.run.smithi026.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2021-02-11T09:42:27.805 INFO:teuthology.orchestra.run.smithi026.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-02-11T09:42:27.806 INFO:teuthology.orchestra.run.smithi026.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-02-11T09:42:27.807 INFO:teuthology.orchestra.run.smithi026.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-02-11T09:42:27.807 INFO:teuthology.orchestra.run.smithi026.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-02-11T09:42:27.844 INFO:teuthology.orchestra.run.smithi133.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-02-11T09:42:27.847 INFO:teuthology.orchestra.run.smithi133.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-02-11T09:42:27.876 INFO:teuthology.orchestra.run.smithi026.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-02-11T09:42:27.878 INFO:teuthology.orchestra.run.smithi026.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-02-11T09:42:27.881 INFO:teuthology.orchestra.run.smithi133.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-02-11T09:42:27.881 INFO:teuthology.orchestra.run.smithi133.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-02-11T09:42:27.881 INFO:teuthology.orchestra.run.smithi133.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2021-02-11T09:42:27.882 INFO:teuthology.orchestra.run.smithi133.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-02-11T09:42:27.883 INFO:teuthology.orchestra.run.smithi133.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-02-11T09:42:27.884 INFO:teuthology.orchestra.run.smithi133.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-02-11T09:42:27.884 INFO:teuthology.orchestra.run.smithi133.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-02-11T09:42:27.955 INFO:teuthology.orchestra.run.smithi026.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-02-11T09:42:27.955 INFO:teuthology.orchestra.run.smithi026.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-02-11T09:42:27.957 INFO:teuthology.orchestra.run.smithi026.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-02-11T09:42:27.957 INFO:teuthology.orchestra.run.smithi026.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-02-11T09:42:27.957 INFO:teuthology.orchestra.run.smithi026.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-02-11T09:42:27.959 INFO:teuthology.orchestra.run.smithi026.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-02-11T09:42:27.961 INFO:teuthology.orchestra.run.smithi026.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-02-11T09:42:27.962 INFO:teuthology.orchestra.run.smithi133.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-02-11T09:42:27.962 INFO:teuthology.orchestra.run.smithi026.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-02-11T09:42:27.964 INFO:teuthology.orchestra.run.smithi133.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-02-11T09:42:27.970 INFO:teuthology.orchestra.run.smithi133.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-02-11T09:42:27.971 INFO:teuthology.orchestra.run.smithi133.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-02-11T09:42:28.032 INFO:teuthology.orchestra.run.smithi026.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-02-11T09:42:28.035 INFO:teuthology.orchestra.run.smithi026.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-02-11T09:42:28.041 INFO:teuthology.orchestra.run.smithi133.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-02-11T09:42:28.042 INFO:teuthology.orchestra.run.smithi133.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-02-11T09:42:28.043 INFO:teuthology.orchestra.run.smithi133.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-02-11T09:42:28.046 INFO:teuthology.orchestra.run.smithi133.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-02-11T09:42:28.049 INFO:teuthology.orchestra.run.smithi133.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-02-11T09:42:28.050 INFO:teuthology.orchestra.run.smithi133.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-02-11T09:42:28.110 INFO:teuthology.orchestra.run.smithi026.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-02-11T09:42:28.116 INFO:teuthology.orchestra.run.smithi026.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-02-11T09:42:28.117 INFO:teuthology.orchestra.run.smithi026.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-02-11T09:42:28.118 INFO:teuthology.orchestra.run.smithi026.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-02-11T09:42:28.118 INFO:teuthology.orchestra.run.smithi026.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-02-11T09:42:28.120 INFO:teuthology.orchestra.run.smithi026.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-02-11T09:42:28.121 INFO:teuthology.orchestra.run.smithi133.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-02-11T09:42:28.124 INFO:teuthology.orchestra.run.smithi026.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-02-11T09:42:28.124 INFO:teuthology.orchestra.run.smithi026.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-02-11T09:42:28.127 INFO:teuthology.orchestra.run.smithi133.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-02-11T09:42:28.135 INFO:teuthology.orchestra.run.smithi133.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-02-11T09:42:28.138 INFO:teuthology.orchestra.run.smithi133.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-02-11T09:42:28.180 INFO:teuthology.orchestra.run.smithi026.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 17.0.0-735-g5e5bb33c-1bionic [5,627 kB] 2021-02-11T09:42:28.188 INFO:teuthology.orchestra.run.smithi026.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-02-11T09:42:28.189 INFO:teuthology.orchestra.run.smithi026.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-02-11T09:42:28.199 INFO:teuthology.orchestra.run.smithi133.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-02-11T09:42:28.200 INFO:teuthology.orchestra.run.smithi133.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-02-11T09:42:28.200 INFO:teuthology.orchestra.run.smithi133.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-02-11T09:42:28.201 INFO:teuthology.orchestra.run.smithi133.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-02-11T09:42:28.203 INFO:teuthology.orchestra.run.smithi133.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-02-11T09:42:28.203 INFO:teuthology.orchestra.run.smithi133.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-02-11T09:42:28.266 INFO:teuthology.orchestra.run.smithi026.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-02-11T09:42:28.279 INFO:teuthology.orchestra.run.smithi133.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-02-11T09:42:28.280 INFO:teuthology.orchestra.run.smithi133.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-02-11T09:42:28.283 INFO:teuthology.orchestra.run.smithi133.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-02-11T09:42:28.289 INFO:teuthology.orchestra.run.smithi026.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-02-11T09:42:28.290 INFO:teuthology.orchestra.run.smithi026.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-02-11T09:42:28.291 INFO:teuthology.orchestra.run.smithi026.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-02-11T09:42:28.302 INFO:teuthology.orchestra.run.smithi133.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-02-11T09:42:28.358 INFO:teuthology.orchestra.run.smithi133.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-02-11T09:42:28.359 INFO:teuthology.orchestra.run.smithi133.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-02-11T09:42:28.426 INFO:teuthology.orchestra.run.smithi026.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-02-11T09:42:28.427 INFO:teuthology.orchestra.run.smithi026.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-02-11T09:42:28.428 INFO:teuthology.orchestra.run.smithi026.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.4 [329 kB] 2021-02-11T09:42:28.435 INFO:teuthology.orchestra.run.smithi026.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-02-11T09:42:28.436 INFO:teuthology.orchestra.run.smithi026.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-02-11T09:42:28.437 INFO:teuthology.orchestra.run.smithi026.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-02-11T09:42:28.438 INFO:teuthology.orchestra.run.smithi026.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-02-11T09:42:28.440 INFO:teuthology.orchestra.run.smithi026.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-02-11T09:42:28.440 INFO:teuthology.orchestra.run.smithi026.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-02-11T09:42:28.453 INFO:teuthology.orchestra.run.smithi026.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 17.0.0-735-g5e5bb33c-1bionic [185 kB] 2021-02-11T09:42:28.458 INFO:teuthology.orchestra.run.smithi026.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 17.0.0-735-g5e5bb33c-1bionic [1,365 kB] 2021-02-11T09:42:28.460 INFO:teuthology.orchestra.run.smithi133.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-02-11T09:42:28.461 INFO:teuthology.orchestra.run.smithi133.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-02-11T09:42:28.461 INFO:teuthology.orchestra.run.smithi133.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.4 [329 kB] 2021-02-11T09:42:28.466 INFO:teuthology.orchestra.run.smithi133.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-02-11T09:42:28.466 INFO:teuthology.orchestra.run.smithi133.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-02-11T09:42:28.467 INFO:teuthology.orchestra.run.smithi133.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-02-11T09:42:28.467 INFO:teuthology.orchestra.run.smithi133.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-02-11T09:42:28.468 INFO:teuthology.orchestra.run.smithi133.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-02-11T09:42:28.516 INFO:teuthology.orchestra.run.smithi133.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-02-11T09:42:28.526 INFO:teuthology.orchestra.run.smithi026.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 17.0.0-735-g5e5bb33c-1bionic [6,661 kB] 2021-02-11T09:42:28.862 INFO:teuthology.orchestra.run.smithi026.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 17.0.0-735-g5e5bb33c-1bionic [24.9 MB] 2021-02-11T09:42:29.762 INFO:teuthology.orchestra.run.smithi133.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 17.0.0-735-g5e5bb33c-1bionic [5,627 kB] 2021-02-11T09:42:45.474 INFO:teuthology.orchestra.run.smithi026.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 17.0.0-735-g5e5bb33c-1bionic [3,808 B] 2021-02-11T09:42:45.475 INFO:teuthology.orchestra.run.smithi026.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 17.0.0-735-g5e5bb33c-1bionic [770 kB] 2021-02-11T09:42:45.475 INFO:teuthology.orchestra.run.smithi026.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 17.0.0-735-g5e5bb33c-1bionic [2,153 kB] 2021-02-11T09:42:45.475 INFO:teuthology.orchestra.run.smithi026.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 17.0.0-735-g5e5bb33c-1bionic [62.2 kB] 2021-02-11T09:42:45.475 INFO:teuthology.orchestra.run.smithi026.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 17.0.0-735-g5e5bb33c-1bionic [67.2 kB] 2021-02-11T09:42:45.475 INFO:teuthology.orchestra.run.smithi026.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 17.0.0-735-g5e5bb33c-1bionic [2,327 kB] 2021-02-11T09:42:45.475 INFO:teuthology.orchestra.run.smithi026.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 17.0.0-735-g5e5bb33c-1bionic [7,885 kB] 2021-02-11T09:42:45.475 INFO:teuthology.orchestra.run.smithi026.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 17.0.0-735-g5e5bb33c-1bionic [3,924 B] 2021-02-11T09:42:45.475 INFO:teuthology.orchestra.run.smithi026.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 17.0.0-735-g5e5bb33c-1bionic [23.3 kB] 2021-02-11T09:42:45.475 INFO:teuthology.orchestra.run.smithi026.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 17.0.0-735-g5e5bb33c-1bionic [45.3 MB] 2021-02-11T09:42:45.475 INFO:teuthology.orchestra.run.smithi026.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 17.0.0-735-g5e5bb33c-1bionic [19.8 kB] 2021-02-11T09:42:45.475 INFO:teuthology.orchestra.run.smithi026.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 17.0.0-735-g5e5bb33c-1bionic [10.3 MB] 2021-02-11T09:42:45.476 INFO:teuthology.orchestra.run.smithi026.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 17.0.0-735-g5e5bb33c-1bionic [77.7 kB] 2021-02-11T09:42:45.476 INFO:teuthology.orchestra.run.smithi026.stdout:Fetched 167 MB in 7s (23.6 MB/s) 2021-02-11T09:42:45.476 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-uritemplate. 2021-02-11T09:42:45.476 INFO:teuthology.orchestra.run.smithi026.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137611 files and directories currently installed.) 2021-02-11T09:42:45.476 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-02-11T09:42:45.476 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-02-11T09:42:45.476 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libjaeger. 2021-02-11T09:42:45.476 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../001-libjaeger_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.476 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libjaeger (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.476 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-02-11T09:42:45.476 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../002-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-02-11T09:42:45.476 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-11T09:42:45.477 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../003-librbd1_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.477 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking librbd1 (17.0.0-735-g5e5bb33c-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-11T09:42:45.477 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../004-librados2_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.477 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking librados2 (17.0.0-735-g5e5bb33c-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-11T09:42:45.477 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libcephfs2. 2021-02-11T09:42:45.477 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../005-libcephfs2_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.477 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libcephfs2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.477 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-rados. 2021-02-11T09:42:45.477 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../006-python3-rados_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.477 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-rados (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.477 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-02-11T09:42:45.477 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../007-python3-ceph-argparse_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:42:45.477 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-ceph-argparse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.478 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-cephfs. 2021-02-11T09:42:45.478 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../008-python3-cephfs_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.478 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-cephfs (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.478 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-ceph-common. 2021-02-11T09:42:45.478 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../009-python3-ceph-common_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:42:45.478 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-ceph-common (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.478 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-prettytable. 2021-02-11T09:42:45.478 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../010-python3-prettytable_0.7.2-3_all.deb ... 2021-02-11T09:42:45.478 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-02-11T09:42:45.478 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-rbd. 2021-02-11T09:42:45.478 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../011-python3-rbd_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.478 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-rbd (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.479 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-02-11T09:42:45.479 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../012-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-02-11T09:42:45.479 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-11T09:42:45.479 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-02-11T09:42:45.479 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../013-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-02-11T09:42:45.479 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-11T09:42:45.479 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package librgw2. 2021-02-11T09:42:45.479 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../014-librgw2_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.479 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking librgw2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.479 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-rgw. 2021-02-11T09:42:45.479 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../015-python3-rgw_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.479 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-rgw (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.480 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-02-11T09:42:45.480 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../016-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-02-11T09:42:45.480 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-11T09:42:45.480 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-02-11T09:42:45.480 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../017-libleveldb1v5_1.20-2_amd64.deb ... 2021-02-11T09:42:45.480 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-02-11T09:42:45.480 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package liboath0. 2021-02-11T09:42:45.480 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../018-liboath0_2.6.1-1_amd64.deb ... 2021-02-11T09:42:45.480 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-02-11T09:42:45.480 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libradosstriper1. 2021-02-11T09:42:45.480 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../019-libradosstriper1_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.480 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libradosstriper1 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.480 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-common. 2021-02-11T09:42:45.481 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../020-ceph-common_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.481 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-common (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.481 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-base. 2021-02-11T09:42:45.481 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../021-ceph-base_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.481 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-base (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.481 INFO:teuthology.orchestra.run.smithi133.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 17.0.0-735-g5e5bb33c-1bionic [185 kB] 2021-02-11T09:42:45.481 INFO:teuthology.orchestra.run.smithi133.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 17.0.0-735-g5e5bb33c-1bionic [1,365 kB] 2021-02-11T09:42:45.481 INFO:teuthology.orchestra.run.smithi133.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 17.0.0-735-g5e5bb33c-1bionic [6,661 kB] 2021-02-11T09:42:45.481 INFO:teuthology.orchestra.run.smithi133.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 17.0.0-735-g5e5bb33c-1bionic [24.9 MB] 2021-02-11T09:42:45.481 INFO:teuthology.orchestra.run.smithi133.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 17.0.0-735-g5e5bb33c-1bionic [3,808 B] 2021-02-11T09:42:45.481 INFO:teuthology.orchestra.run.smithi133.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 17.0.0-735-g5e5bb33c-1bionic [770 kB] 2021-02-11T09:42:45.481 INFO:teuthology.orchestra.run.smithi133.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 17.0.0-735-g5e5bb33c-1bionic [2,153 kB] 2021-02-11T09:42:45.482 INFO:teuthology.orchestra.run.smithi133.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 17.0.0-735-g5e5bb33c-1bionic [62.2 kB] 2021-02-11T09:42:45.482 INFO:teuthology.orchestra.run.smithi133.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 17.0.0-735-g5e5bb33c-1bionic [67.2 kB] 2021-02-11T09:42:45.482 INFO:teuthology.orchestra.run.smithi133.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 17.0.0-735-g5e5bb33c-1bionic [2,327 kB] 2021-02-11T09:42:45.482 INFO:teuthology.orchestra.run.smithi133.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 17.0.0-735-g5e5bb33c-1bionic [7,885 kB] 2021-02-11T09:42:45.482 INFO:teuthology.orchestra.run.smithi133.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 17.0.0-735-g5e5bb33c-1bionic [3,924 B] 2021-02-11T09:42:45.482 INFO:teuthology.orchestra.run.smithi133.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 17.0.0-735-g5e5bb33c-1bionic [23.3 kB] 2021-02-11T09:42:45.482 INFO:teuthology.orchestra.run.smithi133.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 17.0.0-735-g5e5bb33c-1bionic [45.3 MB] 2021-02-11T09:42:45.482 INFO:teuthology.orchestra.run.smithi133.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 17.0.0-735-g5e5bb33c-1bionic [19.8 kB] 2021-02-11T09:42:45.482 INFO:teuthology.orchestra.run.smithi133.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 17.0.0-735-g5e5bb33c-1bionic [10.3 MB] 2021-02-11T09:42:45.482 INFO:teuthology.orchestra.run.smithi133.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 17.0.0-735-g5e5bb33c-1bionic [77.7 kB] 2021-02-11T09:42:45.482 INFO:teuthology.orchestra.run.smithi133.stdout:Fetched 167 MB in 9s (17.7 MB/s) 2021-02-11T09:42:45.482 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-uritemplate. 2021-02-11T09:42:45.483 INFO:teuthology.orchestra.run.smithi133.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137611 files and directories currently installed.) 2021-02-11T09:42:45.483 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-02-11T09:42:45.483 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-02-11T09:42:45.483 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libjaeger. 2021-02-11T09:42:45.483 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../001-libjaeger_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.483 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libjaeger (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.483 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-02-11T09:42:45.483 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../002-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-02-11T09:42:45.483 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-11T09:42:45.483 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../003-librbd1_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.483 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking librbd1 (17.0.0-735-g5e5bb33c-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-11T09:42:45.483 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../004-librados2_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.483 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking librados2 (17.0.0-735-g5e5bb33c-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-11T09:42:45.484 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libcephfs2. 2021-02-11T09:42:45.484 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../005-libcephfs2_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.484 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libcephfs2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.484 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-rados. 2021-02-11T09:42:45.484 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../006-python3-rados_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.484 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-rados (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.484 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-02-11T09:42:45.484 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../007-python3-ceph-argparse_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:42:45.484 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-ceph-argparse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.484 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-cephfs. 2021-02-11T09:42:45.484 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../008-python3-cephfs_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.484 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-cephfs (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.485 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-ceph-common. 2021-02-11T09:42:45.485 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../009-python3-ceph-common_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:42:45.485 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-ceph-common (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.485 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-prettytable. 2021-02-11T09:42:45.485 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../010-python3-prettytable_0.7.2-3_all.deb ... 2021-02-11T09:42:45.485 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-02-11T09:42:45.485 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-rbd. 2021-02-11T09:42:45.485 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../011-python3-rbd_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.485 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-rbd (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.485 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-02-11T09:42:45.485 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../012-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-02-11T09:42:45.485 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-11T09:42:45.485 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-02-11T09:42:45.485 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../013-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-02-11T09:42:45.486 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-11T09:42:45.486 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package librgw2. 2021-02-11T09:42:45.486 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../014-librgw2_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.486 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking librgw2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.486 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-rgw. 2021-02-11T09:42:45.486 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../015-python3-rgw_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.486 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-rgw (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.486 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-02-11T09:42:45.486 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../016-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-02-11T09:42:45.486 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-11T09:42:45.486 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-02-11T09:42:45.486 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../017-libleveldb1v5_1.20-2_amd64.deb ... 2021-02-11T09:42:45.487 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-02-11T09:42:45.487 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package liboath0. 2021-02-11T09:42:45.487 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../018-liboath0_2.6.1-1_amd64.deb ... 2021-02-11T09:42:45.487 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-02-11T09:42:45.487 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libradosstriper1. 2021-02-11T09:42:45.487 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../019-libradosstriper1_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.487 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libradosstriper1 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.487 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-common. 2021-02-11T09:42:45.487 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../020-ceph-common_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:45.487 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-common (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:45.753 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-dateutil. 2021-02-11T09:42:45.767 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../022-python3-dateutil_2.6.1-1_all.deb ... 2021-02-11T09:42:45.803 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-02-11T09:42:46.148 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-asn1crypto. 2021-02-11T09:42:46.164 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../023-python3-asn1crypto_0.24.0-1_all.deb ... 2021-02-11T09:42:46.206 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-02-11T09:42:46.476 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-cffi-backend. 2021-02-11T09:42:46.490 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../024-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-02-11T09:42:46.534 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-02-11T09:42:46.662 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-base. 2021-02-11T09:42:46.676 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../021-ceph-base_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:46.713 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-base (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:46.870 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-cryptography. 2021-02-11T09:42:46.883 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../025-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-02-11T09:42:46.920 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-11T09:42:47.310 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-openssl. 2021-02-11T09:42:47.325 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../026-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-02-11T09:42:47.373 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-02-11T09:42:47.617 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-dateutil. 2021-02-11T09:42:47.626 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-02-11T09:42:47.631 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../022-python3-dateutil_2.6.1-1_all.deb ... 2021-02-11T09:42:47.639 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:42:47.675 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-02-11T09:42:47.676 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr-modules-core (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:47.986 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-asn1crypto. 2021-02-11T09:42:48.000 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../023-python3-asn1crypto_0.24.0-1_all.deb ... 2021-02-11T09:42:48.036 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-02-11T09:42:48.047 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-bcrypt. 2021-02-11T09:42:48.062 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-02-11T09:42:48.106 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-02-11T09:42:48.289 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-cffi-backend. 2021-02-11T09:42:48.302 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../024-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-02-11T09:42:48.331 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-02-11T09:42:48.400 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-cherrypy3. 2021-02-11T09:42:48.415 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-2_all.deb ... 2021-02-11T09:42:48.473 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-02-11T09:42:48.633 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-cryptography. 2021-02-11T09:42:48.647 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../025-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-02-11T09:42:48.691 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-11T09:42:48.803 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-jwt. 2021-02-11T09:42:48.819 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../030-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-02-11T09:42:48.861 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-02-11T09:42:49.078 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-openssl. 2021-02-11T09:42:49.092 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../026-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-02-11T09:42:49.145 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-02-11T09:42:49.215 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-logutils. 2021-02-11T09:42:49.231 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../031-python3-logutils_0.3.3-5_all.deb ... 2021-02-11T09:42:49.273 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-02-11T09:42:49.398 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-02-11T09:42:49.411 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:42:49.456 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr-modules-core (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:49.576 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-markupsafe. 2021-02-11T09:42:49.591 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../032-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-02-11T09:42:49.659 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-02-11T09:42:49.819 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-bcrypt. 2021-02-11T09:42:49.833 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-02-11T09:42:49.877 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-02-11T09:42:49.987 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-mako. 2021-02-11T09:42:50.002 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../033-python3-mako_1.0.7+ds1-1_all.deb ... 2021-02-11T09:42:50.053 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-02-11T09:42:50.163 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-cherrypy3. 2021-02-11T09:42:50.178 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-2_all.deb ... 2021-02-11T09:42:50.221 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-02-11T09:42:50.415 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-simplegeneric. 2021-02-11T09:42:50.428 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../034-python3-simplegeneric_0.8.1-1_all.deb ... 2021-02-11T09:42:50.465 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-02-11T09:42:50.541 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-jwt. 2021-02-11T09:42:50.555 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../030-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-02-11T09:42:50.600 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-02-11T09:42:50.768 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-singledispatch. 2021-02-11T09:42:50.781 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../035-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-02-11T09:42:50.834 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-02-11T09:42:50.919 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-logutils. 2021-02-11T09:42:50.934 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../031-python3-logutils_0.3.3-5_all.deb ... 2021-02-11T09:42:50.986 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-02-11T09:42:51.119 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-webob. 2021-02-11T09:42:51.135 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../036-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-02-11T09:42:51.178 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-02-11T09:42:51.289 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-markupsafe. 2021-02-11T09:42:51.303 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../032-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-02-11T09:42:51.347 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-02-11T09:42:51.541 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-bs4. 2021-02-11T09:42:51.555 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../037-python3-bs4_4.6.0-1_all.deb ... 2021-02-11T09:42:51.607 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-02-11T09:42:51.650 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-mako. 2021-02-11T09:42:51.663 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../033-python3-mako_1.0.7+ds1-1_all.deb ... 2021-02-11T09:42:51.708 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-02-11T09:42:51.960 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-waitress. 2021-02-11T09:42:51.976 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../038-python3-waitress_1.0.1-1_all.deb ... 2021-02-11T09:42:52.019 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-02-11T09:42:52.053 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-simplegeneric. 2021-02-11T09:42:52.067 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../034-python3-simplegeneric_0.8.1-1_all.deb ... 2021-02-11T09:42:52.111 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-02-11T09:42:52.338 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-tempita. 2021-02-11T09:42:52.354 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../039-python3-tempita_0.5.2-2_all.deb ... 2021-02-11T09:42:52.397 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-02-11T09:42:52.412 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-singledispatch. 2021-02-11T09:42:52.426 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../035-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-02-11T09:42:52.470 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-02-11T09:42:52.725 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-paste. 2021-02-11T09:42:52.740 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../040-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-02-11T09:42:52.757 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-webob. 2021-02-11T09:42:52.771 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../036-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-02-11T09:42:52.783 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-11T09:42:52.815 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-02-11T09:42:53.078 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-02-11T09:42:53.091 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../041-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-02-11T09:42:53.127 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-bs4. 2021-02-11T09:42:53.142 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../037-python3-bs4_4.6.0-1_all.deb ... 2021-02-11T09:42:53.161 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-02-11T09:42:53.185 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-02-11T09:42:53.464 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pastedeploy. 2021-02-11T09:42:53.478 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../042-python3-pastedeploy_1.5.2-4_all.deb ... 2021-02-11T09:42:53.488 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-waitress. 2021-02-11T09:42:53.503 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../038-python3-waitress_1.0.1-1_all.deb ... 2021-02-11T09:42:53.522 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-02-11T09:42:53.579 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-02-11T09:42:53.842 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-webtest. 2021-02-11T09:42:53.855 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../043-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-02-11T09:42:53.900 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-02-11T09:42:53.908 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-tempita. 2021-02-11T09:42:53.922 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../039-python3-tempita_0.5.2-2_all.deb ... 2021-02-11T09:42:53.966 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-02-11T09:42:54.253 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pecan. 2021-02-11T09:42:54.267 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../044-python3-pecan_1.2.1-2_all.deb ... 2021-02-11T09:42:54.269 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-paste. 2021-02-11T09:42:54.283 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../040-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-02-11T09:42:54.327 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-11T09:42:54.327 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-02-11T09:42:54.630 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-02-11T09:42:54.645 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../041-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-02-11T09:42:54.688 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-02-11T09:42:54.699 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-werkzeug. 2021-02-11T09:42:54.716 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../045-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-02-11T09:42:54.758 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-11T09:42:54.983 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pastedeploy. 2021-02-11T09:42:54.998 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../042-python3-pastedeploy_1.5.2-4_all.deb ... 2021-02-11T09:42:55.041 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-02-11T09:42:55.136 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr. 2021-02-11T09:42:55.150 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../046-ceph-mgr_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:55.194 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:55.386 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-webtest. 2021-02-11T09:42:55.399 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../043-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-02-11T09:42:55.444 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-02-11T09:42:55.681 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mon. 2021-02-11T09:42:55.695 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../047-ceph-mon_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:55.755 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mon (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:55.771 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pecan. 2021-02-11T09:42:55.786 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../044-python3-pecan_1.2.1-2_all.deb ... 2021-02-11T09:42:55.829 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-02-11T09:42:56.185 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-werkzeug. 2021-02-11T09:42:56.201 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../045-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-02-11T09:42:56.243 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-11T09:42:56.596 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr. 2021-02-11T09:42:56.611 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../046-ceph-mgr_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:56.671 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:57.124 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mon. 2021-02-11T09:42:57.138 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../047-ceph-mon_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:57.182 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mon (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:57.426 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-osd. 2021-02-11T09:42:57.440 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../048-ceph-osd_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:57.485 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-osd (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:58.820 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-osd. 2021-02-11T09:42:58.834 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../048-ceph-osd_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:58.878 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-osd (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:59.525 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph. 2021-02-11T09:42:59.541 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../049-ceph_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:42:59.583 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:59.936 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-fuse. 2021-02-11T09:43:03.284 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph. 2021-02-11T09:43:03.284 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../049-ceph_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:43:03.284 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:03.284 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-fuse. 2021-02-11T09:43:03.284 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../050-ceph-fuse_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:43:03.284 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-fuse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:03.285 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mds. 2021-02-11T09:43:03.285 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../051-ceph-mds_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:43:03.285 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mds (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:03.285 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package cephadm. 2021-02-11T09:43:03.285 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../052-cephadm_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:43:03.285 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking cephadm (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:03.285 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-jinja2. 2021-02-11T09:43:03.285 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../053-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-02-11T09:43:03.285 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-11T09:43:03.285 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../050-ceph-fuse_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:43:03.285 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-fuse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:03.285 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mds. 2021-02-11T09:43:03.286 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../051-ceph-mds_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:43:03.286 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mds (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:03.286 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package cephadm. 2021-02-11T09:43:03.286 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../052-cephadm_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:43:03.286 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking cephadm (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:03.286 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-jinja2. 2021-02-11T09:43:03.286 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../053-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-02-11T09:43:03.286 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-11T09:43:03.286 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-02-11T09:43:03.286 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../054-ceph-mgr-cephadm_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:43:03.286 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr-cephadm (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:03.286 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-repoze.lru. 2021-02-11T09:43:03.286 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../055-python3-repoze.lru_0.7-1_all.deb ... 2021-02-11T09:43:03.286 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-02-11T09:43:03.287 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-routes. 2021-02-11T09:43:03.287 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../056-python3-routes_2.4.1-1_all.deb ... 2021-02-11T09:43:03.287 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-02-11T09:43:03.327 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-02-11T09:43:03.342 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:43:03.386 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr-dashboard (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:03.393 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-02-11T09:43:03.407 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../054-ceph-mgr-cephadm_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:43:03.452 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr-cephadm (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:03.771 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-repoze.lru. 2021-02-11T09:43:03.784 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../055-python3-repoze.lru_0.7-1_all.deb ... 2021-02-11T09:43:03.821 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-02-11T09:43:04.125 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-routes. 2021-02-11T09:43:04.139 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../056-python3-routes_2.4.1-1_all.deb ... 2021-02-11T09:43:04.183 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-02-11T09:43:04.308 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-numpy. 2021-02-11T09:43:04.325 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-02-11T09:43:04.374 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-11T09:43:04.519 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-02-11T09:43:04.536 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:43:04.578 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr-dashboard (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:04.972 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-decorator. 2021-02-11T09:43:04.987 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-02-11T09:43:05.046 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-02-11T09:43:05.333 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-scipy. 2021-02-11T09:43:05.349 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-02-11T09:43:05.391 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-02-11T09:43:05.482 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-numpy. 2021-02-11T09:43:05.497 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-02-11T09:43:05.540 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-11T09:43:06.180 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-decorator. 2021-02-11T09:43:06.193 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-02-11T09:43:06.230 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-02-11T09:43:06.549 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-scipy. 2021-02-11T09:43:06.563 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-02-11T09:43:06.607 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-02-11T09:43:06.760 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-02-11T09:43:06.775 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-02-11T09:43:06.818 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-02-11T09:43:07.282 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-joblib. 2021-02-11T09:43:07.296 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-02-11T09:43:07.340 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-joblib (0.11-1) ... 2021-02-11T09:43:07.722 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-sklearn. 2021-02-11T09:43:07.736 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-02-11T09:43:07.789 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-02-11T09:43:08.018 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-02-11T09:43:08.034 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-02-11T09:43:08.076 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-02-11T09:43:08.364 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-02-11T09:43:08.377 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:43:08.422 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:08.533 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-joblib. 2021-02-11T09:43:08.549 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-02-11T09:43:08.592 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-joblib (0.11-1) ... 2021-02-11T09:43:08.929 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-sklearn. 2021-02-11T09:43:08.943 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-02-11T09:43:08.987 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-02-11T09:43:09.505 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-02-11T09:43:09.521 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:43:09.563 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:09.564 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-monotonic. 2021-02-11T09:43:09.579 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-02-11T09:43:09.622 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-02-11T09:43:09.950 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-fasteners. 2021-02-11T09:43:09.964 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-02-11T09:43:10.008 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-02-11T09:43:10.311 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-httplib2. 2021-02-11T09:43:10.326 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2021-02-11T09:43:10.369 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-11T09:43:10.720 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-monotonic. 2021-02-11T09:43:10.730 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pyasn1. 2021-02-11T09:43:10.735 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-02-11T09:43:10.746 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-02-11T09:43:10.778 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-02-11T09:43:10.797 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-02-11T09:43:11.081 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-fasteners. 2021-02-11T09:43:11.095 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-02-11T09:43:11.140 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-02-11T09:43:11.159 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-02-11T09:43:11.173 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-02-11T09:43:11.217 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-11T09:43:11.451 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-httplib2. 2021-02-11T09:43:11.466 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2021-02-11T09:43:11.509 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-11T09:43:11.528 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-rsa. 2021-02-11T09:43:11.543 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-02-11T09:43:11.603 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-02-11T09:43:11.820 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pyasn1. 2021-02-11T09:43:11.836 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-02-11T09:43:11.878 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-02-11T09:43:11.923 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-02-11T09:43:11.939 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-02-11T09:43:11.981 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-11T09:43:12.227 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-02-11T09:43:12.243 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-02-11T09:43:12.290 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-11T09:43:12.536 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-oauth2client. 2021-02-11T09:43:12.551 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-02-11T09:43:12.594 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-02-11T09:43:12.609 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-rsa. 2021-02-11T09:43:12.625 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-02-11T09:43:12.667 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-02-11T09:43:12.939 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-setuptools. 2021-02-11T09:43:12.955 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-02-11T09:43:12.987 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-02-11T09:43:12.997 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-02-11T09:43:13.002 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-02-11T09:43:13.045 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-11T09:43:13.417 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-websocket. 2021-02-11T09:43:13.432 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-02-11T09:43:13.441 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-oauth2client. 2021-02-11T09:43:13.456 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-02-11T09:43:13.476 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-02-11T09:43:13.491 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-02-11T09:43:13.779 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-kubernetes. 2021-02-11T09:43:13.794 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-setuptools. 2021-02-11T09:43:13.796 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-02-11T09:43:13.809 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-02-11T09:43:13.837 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-11T09:43:13.844 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-02-11T09:43:14.189 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-websocket. 2021-02-11T09:43:14.203 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-02-11T09:43:14.239 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-02-11T09:43:14.308 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-02-11T09:43:14.325 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:43:14.366 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:14.492 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-kubernetes. 2021-02-11T09:43:14.509 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-02-11T09:43:14.542 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-11T09:43:14.719 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-json-pointer. 2021-02-11T09:43:14.732 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-02-11T09:43:14.777 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-02-11T09:43:14.938 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-02-11T09:43:14.953 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:43:15.004 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:15.113 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-jsonpatch. 2021-02-11T09:43:15.129 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-02-11T09:43:15.200 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-11T09:43:15.348 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-json-pointer. 2021-02-11T09:43:15.363 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-02-11T09:43:15.406 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-02-11T09:43:15.527 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-02-11T09:43:15.542 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../079-ceph-mgr-rook_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:43:15.585 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-mgr-rook (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:15.768 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-jsonpatch. 2021-02-11T09:43:15.783 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-02-11T09:43:15.829 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-11T09:43:15.930 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libonig4:amd64. 2021-02-11T09:43:15.944 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-02-11T09:43:16.074 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-02-11T09:43:16.123 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-02-11T09:43:16.138 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../079-ceph-mgr-rook_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:43:16.181 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr-rook (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:16.477 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libjq1:amd64. 2021-02-11T09:43:16.491 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-02-11T09:43:16.492 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libonig4:amd64. 2021-02-11T09:43:16.509 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-02-11T09:43:16.535 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-02-11T09:43:16.636 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-02-11T09:43:16.813 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package jq. 2021-02-11T09:43:16.827 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-02-11T09:43:16.871 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-02-11T09:43:17.006 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libjq1:amd64. 2021-02-11T09:43:17.019 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-02-11T09:43:17.072 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-02-11T09:43:17.148 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package socat. 2021-02-11T09:43:17.165 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-02-11T09:43:17.206 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-02-11T09:43:17.342 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package jq. 2021-02-11T09:43:17.355 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-02-11T09:43:17.400 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-02-11T09:43:17.644 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package socat. 2021-02-11T09:43:17.652 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-02-11T09:43:17.659 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-02-11T09:43:17.667 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-02-11T09:43:17.702 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-02-11T09:43:17.710 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-11T09:43:17.997 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package xmlstarlet. 2021-02-11T09:43:18.013 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-02-11T09:43:18.055 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-02-11T09:43:18.122 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-02-11T09:43:18.138 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-02-11T09:43:18.180 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-11T09:43:18.442 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package ceph-test. 2021-02-11T09:43:18.459 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../086-ceph-test_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:43:18.509 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package xmlstarlet. 2021-02-11T09:43:18.509 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking ceph-test (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:18.523 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-02-11T09:43:18.575 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-02-11T09:43:18.946 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-test. 2021-02-11T09:43:18.959 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../086-ceph-test_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:43:19.012 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-test (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:21.930 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package libcephfs-dev. 2021-02-11T09:43:21.945 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../087-libcephfs-dev_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:43:21.988 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking libcephfs-dev (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:22.282 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package nvme-cli. 2021-02-11T09:43:22.298 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-02-11T09:43:22.341 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-02-11T09:43:22.433 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libcephfs-dev. 2021-02-11T09:43:22.449 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../087-libcephfs-dev_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:43:22.491 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libcephfs-dev (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:22.687 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python-attr. 2021-02-11T09:43:22.704 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-02-11T09:43:22.745 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python-attr (17.4.0-2) ... 2021-02-11T09:43:22.777 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package nvme-cli. 2021-02-11T09:43:22.792 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-02-11T09:43:22.835 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-02-11T09:43:23.073 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python-funcsigs. 2021-02-11T09:43:23.087 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-02-11T09:43:23.131 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-02-11T09:43:23.206 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python-attr. 2021-02-11T09:43:23.220 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-02-11T09:43:23.264 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python-attr (17.4.0-2) ... 2021-02-11T09:43:23.459 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python-pluggy. 2021-02-11T09:43:23.472 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-02-11T09:43:23.517 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-02-11T09:43:23.568 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python-funcsigs. 2021-02-11T09:43:23.583 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-02-11T09:43:23.642 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-02-11T09:43:23.879 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python-py. 2021-02-11T09:43:23.894 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-02-11T09:43:23.937 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python-pluggy. 2021-02-11T09:43:23.945 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python-py (1.5.2-1) ... 2021-02-11T09:43:23.951 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-02-11T09:43:23.996 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-02-11T09:43:24.315 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python-pytest. 2021-02-11T09:43:24.330 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-02-11T09:43:24.332 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python-py. 2021-02-11T09:43:24.349 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-02-11T09:43:24.373 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-02-11T09:43:24.407 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python-py (1.5.2-1) ... 2021-02-11T09:43:24.736 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-attr. 2021-02-11T09:43:24.753 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-02-11T09:43:24.761 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python-pytest. 2021-02-11T09:43:24.775 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-02-11T09:43:24.802 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-02-11T09:43:24.819 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-02-11T09:43:25.130 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-cycler. 2021-02-11T09:43:25.144 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-02-11T09:43:25.155 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-attr. 2021-02-11T09:43:25.170 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-02-11T09:43:25.205 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-02-11T09:43:25.213 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-02-11T09:43:25.519 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-cycler. 2021-02-11T09:43:25.535 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-02-11T09:43:25.585 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-02-11T09:43:25.608 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-webencodings. 2021-02-11T09:43:25.624 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-02-11T09:43:25.666 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-02-11T09:43:25.863 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-webencodings. 2021-02-11T09:43:25.880 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-02-11T09:43:25.921 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-02-11T09:43:25.994 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-html5lib. 2021-02-11T09:43:26.010 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-02-11T09:43:26.061 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-02-11T09:43:26.240 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-html5lib. 2021-02-11T09:43:26.255 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-02-11T09:43:26.298 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-02-11T09:43:26.473 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-02-11T09:43:26.488 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-02-11T09:43:26.539 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-11T09:43:26.627 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-02-11T09:43:26.640 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-02-11T09:43:26.685 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-11T09:43:26.968 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pyparsing. 2021-02-11T09:43:26.984 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-02-11T09:43:27.034 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-11T09:43:27.088 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pyparsing. 2021-02-11T09:43:27.104 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-02-11T09:43:27.146 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-11T09:43:27.371 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-tz. 2021-02-11T09:43:27.385 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-02-11T09:43:27.429 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-tz (2018.3-2) ... 2021-02-11T09:43:27.465 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-tz. 2021-02-11T09:43:27.479 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-02-11T09:43:27.523 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-tz (2018.3-2) ... 2021-02-11T09:43:27.757 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-matplotlib. 2021-02-11T09:43:27.771 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-02-11T09:43:27.815 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-11T09:43:27.852 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-matplotlib. 2021-02-11T09:43:27.867 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-02-11T09:43:27.910 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-11T09:43:28.523 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-olefile. 2021-02-11T09:43:28.539 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-02-11T09:43:28.557 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-olefile. 2021-02-11T09:43:28.571 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-02-11T09:43:28.581 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-02-11T09:43:28.615 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-02-11T09:43:28.926 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pastescript. 2021-02-11T09:43:28.940 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-02-11T09:43:28.958 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pastescript. 2021-02-11T09:43:28.974 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-02-11T09:43:28.985 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-02-11T09:43:29.025 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-02-11T09:43:29.388 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pil:amd64. 2021-02-11T09:43:29.389 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pil:amd64. 2021-02-11T09:43:29.402 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.4_amd64.deb ... 2021-02-11T09:43:29.403 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.4_amd64.deb ... 2021-02-11T09:43:29.446 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-11T09:43:29.455 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-11T09:43:29.817 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pluggy. 2021-02-11T09:43:29.832 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pluggy. 2021-02-11T09:43:29.833 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-02-11T09:43:29.849 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-02-11T09:43:29.875 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-02-11T09:43:29.890 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-02-11T09:43:30.178 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-py. 2021-02-11T09:43:30.191 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-02-11T09:43:30.210 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-py. 2021-02-11T09:43:30.224 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-02-11T09:43:30.236 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-py (1.5.2-1) ... 2021-02-11T09:43:30.268 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-py (1.5.2-1) ... 2021-02-11T09:43:30.539 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pyinotify. 2021-02-11T09:43:44.281 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pyinotify. 2021-02-11T09:43:44.281 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-02-11T09:43:44.281 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-02-11T09:43:44.281 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-pytest. 2021-02-11T09:43:44.281 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-02-11T09:43:44.281 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-02-11T09:43:44.281 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-simplejson. 2021-02-11T09:43:44.281 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-02-11T09:43:44.281 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-02-11T09:43:44.282 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package python3-tk:amd64. 2021-02-11T09:43:44.282 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-02-11T09:43:44.282 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-11T09:43:44.282 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package radosgw. 2021-02-11T09:43:44.282 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../111-radosgw_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:43:44.282 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking radosgw (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:44.282 INFO:teuthology.orchestra.run.smithi026.stdout:Selecting previously unselected package rbd-fuse. 2021-02-11T09:43:44.282 INFO:teuthology.orchestra.run.smithi026.stdout:Preparing to unpack .../112-rbd-fuse_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:43:44.282 INFO:teuthology.orchestra.run.smithi026.stdout:Unpacking rbd-fuse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:44.282 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-02-11T09:43:44.282 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-11T09:43:44.282 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-11T09:43:44.283 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-11T09:43:44.283 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-11T09:43:44.283 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-02-11T09:43:44.283 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-webencodings (0.5-2) ... 2021-02-11T09:43:44.283 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python-attr (17.4.0-2) ... 2021-02-11T09:43:44.283 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-02-11T09:43:44.283 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-py (1.5.2-1) ... 2021-02-11T09:43:44.283 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-02-11T09:43:44.283 INFO:teuthology.orchestra.run.smithi026.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-02-11T09:43:44.283 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-02-11T09:43:44.283 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-ceph-argparse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:44.283 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-02-11T09:43:44.283 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-02-11T09:43:44.284 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-11T09:43:44.284 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-02-11T09:43:44.284 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-02-11T09:43:44.284 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python-py (1.5.2-1) ... 2021-02-11T09:43:44.284 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-02-11T09:43:44.284 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-02-11T09:43:44.284 INFO:teuthology.orchestra.run.smithi026.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-02-11T09:43:44.284 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-02-11T09:43:44.284 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-02-11T09:43:44.284 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-02-11T09:43:44.284 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-11T09:43:44.284 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-02-11T09:43:44.285 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-02-11T09:43:44.285 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-02-11T09:43:44.285 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-02-11T09:43:44.285 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up cephadm (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:44.285 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-02-11T09:43:44.285 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-02-11T09:43:44.285 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pytest. 2021-02-11T09:43:44.285 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-02-11T09:43:44.285 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-02-11T09:43:44.285 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-simplejson. 2021-02-11T09:43:44.285 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-02-11T09:43:44.285 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-02-11T09:43:44.286 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-tk:amd64. 2021-02-11T09:43:44.286 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-02-11T09:43:44.286 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-11T09:43:44.286 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package radosgw. 2021-02-11T09:43:44.286 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../111-radosgw_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:43:44.286 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking radosgw (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:44.286 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package rbd-fuse. 2021-02-11T09:43:44.286 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../112-rbd-fuse_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:43:44.286 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking rbd-fuse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:44.286 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-02-11T09:43:44.286 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-11T09:43:44.286 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-11T09:43:44.287 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-11T09:43:44.287 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-11T09:43:44.287 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-02-11T09:43:44.287 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-webencodings (0.5-2) ... 2021-02-11T09:43:44.287 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python-attr (17.4.0-2) ... 2021-02-11T09:43:44.287 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-02-11T09:43:44.287 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-py (1.5.2-1) ... 2021-02-11T09:43:44.287 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-02-11T09:43:44.287 INFO:teuthology.orchestra.run.smithi133.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-02-11T09:43:44.287 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-02-11T09:43:44.287 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-ceph-argparse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:44.287 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-02-11T09:43:44.287 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-02-11T09:43:44.288 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-11T09:43:44.288 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-02-11T09:43:44.288 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-02-11T09:43:44.288 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python-py (1.5.2-1) ... 2021-02-11T09:43:44.288 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-02-11T09:43:44.288 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-02-11T09:43:44.288 INFO:teuthology.orchestra.run.smithi133.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-02-11T09:43:44.288 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-02-11T09:43:44.288 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-02-11T09:43:44.288 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-02-11T09:43:44.288 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-11T09:43:44.288 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-02-11T09:43:44.288 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-02-11T09:43:44.288 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-02-11T09:43:44.289 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-02-11T09:43:44.289 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up cephadm (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:44.424 INFO:teuthology.orchestra.run.smithi133.stdout:Adding system user cephadm....done 2021-02-11T09:43:44.613 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-02-11T09:43:44.859 INFO:teuthology.orchestra.run.smithi026.stdout:Adding system user cephadm....done 2021-02-11T09:43:44.905 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-11T09:43:45.031 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-02-11T09:43:45.032 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-02-11T09:43:45.183 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-02-11T09:43:45.333 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-11T09:43:45.475 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-02-11T09:43:45.526 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-02-11T09:43:45.643 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-02-11T09:43:45.810 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-02-11T09:43:45.936 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-02-11T09:43:45.986 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-02-11T09:43:46.271 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-ceph-common (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:46.287 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-02-11T09:43:46.413 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-02-11T09:43:46.589 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-monotonic (1.1-2) ... 2021-02-11T09:43:46.765 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-ceph-common (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:46.865 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-11T09:43:46.982 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-11T09:43:47.083 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-monotonic (1.1-2) ... 2021-02-11T09:43:47.334 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-02-11T09:43:47.384 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-11T09:43:47.518 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-11T09:43:47.693 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-11T09:43:47.783 INFO:teuthology.orchestra.run.smithi133.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-02-11T09:43:47.807 INFO:teuthology.orchestra.run.smithi133.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-02-11T09:43:47.894 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-02-11T09:43:48.037 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-02-11T09:43:48.254 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-11T09:43:48.305 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-11T09:43:48.344 INFO:teuthology.orchestra.run.smithi026.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-02-11T09:43:48.377 INFO:teuthology.orchestra.run.smithi026.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-02-11T09:43:48.581 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-11T09:43:48.615 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-02-11T09:43:48.898 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-attr (17.4.0-2) ... 2021-02-11T09:43:48.916 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-11T09:43:49.225 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-11T09:43:49.226 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-02-11T09:43:49.427 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libjaeger (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:49.537 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-11T09:43:49.619 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-attr (17.4.0-2) ... 2021-02-11T09:43:49.913 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-02-11T09:43:49.938 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-02-11T09:43:49.986 INFO:teuthology.orchestra.run.smithi133.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-02-11T09:43:50.139 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libjaeger (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:50.223 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up liboath0 (2.6.1-1) ... 2021-02-11T09:43:50.298 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-11T09:43:50.375 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-11T09:43:50.501 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-11T09:43:50.700 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-02-11T09:43:50.786 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-02-11T09:43:50.831 INFO:teuthology.orchestra.run.smithi026.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-02-11T09:43:50.895 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-02-11T09:43:51.077 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up liboath0 (2.6.1-1) ... 2021-02-11T09:43:51.263 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-11T09:43:51.263 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-11T09:43:51.365 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-02-11T09:43:51.414 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-11T09:43:51.641 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-tz (2018.3-2) ... 2021-02-11T09:43:51.665 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-02-11T09:43:51.791 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-02-11T09:43:51.951 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-02-11T09:43:52.201 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-11T09:43:52.270 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-02-11T09:43:52.336 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-02-11T09:43:52.562 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-joblib (0.11-1) ... 2021-02-11T09:43:52.646 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-tz (2018.3-2) ... 2021-02-11T09:43:52.930 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-routes (2.4.1-1) ... 2021-02-11T09:43:52.956 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-02-11T09:43:53.231 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-02-11T09:43:53.257 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-02-11T09:43:53.507 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-02-11T09:43:53.567 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-joblib (0.11-1) ... 2021-02-11T09:43:53.825 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-11T09:43:53.977 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-routes (2.4.1-1) ... 2021-02-11T09:43:54.160 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-02-11T09:43:54.295 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-02-11T09:43:54.462 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up jq (1.5+dfsg-2) ... 2021-02-11T09:43:54.588 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-02-11T09:43:54.604 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-02-11T09:43:54.939 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-11T09:43:55.324 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-02-11T09:43:55.634 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up jq (1.5+dfsg-2) ... 2021-02-11T09:43:55.735 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-11T09:43:55.785 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-02-11T09:43:56.053 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-02-11T09:43:56.379 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-02-11T09:43:56.481 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-02-11T09:43:56.824 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python-pytest (3.3.2-2) ... 2021-02-11T09:43:56.983 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-11T09:43:57.075 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-02-11T09:43:57.176 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-02-11T09:43:57.277 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up librados2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:57.309 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-02-11T09:43:57.394 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-02-11T09:43:57.652 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-02-11T09:43:57.704 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libcephfs2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:57.820 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-02-11T09:43:57.829 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libcephfs-dev (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:57.930 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-rados (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:58.031 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-02-11T09:43:58.196 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python-pytest (3.3.2-2) ... 2021-02-11T09:43:58.324 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-02-11T09:43:58.456 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-02-11T09:43:58.582 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-02-11T09:43:58.642 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-02-11T09:43:58.708 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up librados2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:58.834 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-02-11T09:43:59.037 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-11T09:43:59.185 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libcephfs2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:59.312 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libcephfs-dev (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:59.437 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-rados (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:43:59.589 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-02-11T09:43:59.881 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-02-11T09:43:59.915 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-02-11T09:44:00.216 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-02-11T09:44:00.635 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-11T09:44:00.711 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-02-11T09:44:01.004 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libradosstriper1 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:01.129 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-cephfs (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:01.239 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up librgw2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.655 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-02-11T09:44:06.655 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-02-11T09:44:06.655 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up libradosstriper1 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.655 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-cephfs (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.655 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up librgw2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.655 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-fuse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.655 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-11T09:44:06.655 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-11T09:44:06.655 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up librbd1 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.655 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-02-11T09:44:06.656 INFO:teuthology.orchestra.run.smithi026.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-02-11T09:44:06.656 INFO:teuthology.orchestra.run.smithi026.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-02-11T09:44:06.656 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-11T09:44:06.656 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr-modules-core (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.656 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-02-11T09:44:06.656 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-rgw (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.656 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up rbd-fuse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.656 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up python3-rbd (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.656 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-common (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.656 INFO:teuthology.orchestra.run.smithi026.stdout:Adding group ceph....done 2021-02-11T09:44:06.656 INFO:teuthology.orchestra.run.smithi026.stdout:Adding system user ceph....done 2021-02-11T09:44:06.656 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-fuse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.656 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-11T09:44:06.656 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-11T09:44:06.657 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up librbd1 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.657 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-02-11T09:44:06.657 INFO:teuthology.orchestra.run.smithi133.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-02-11T09:44:06.657 INFO:teuthology.orchestra.run.smithi133.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-02-11T09:44:06.657 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-11T09:44:06.657 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr-modules-core (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.657 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-02-11T09:44:06.657 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-rgw (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.657 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up rbd-fuse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.657 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-rbd (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.657 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-common (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.657 INFO:teuthology.orchestra.run.smithi133.stdout:Adding group ceph....done 2021-02-11T09:44:06.657 INFO:teuthology.orchestra.run.smithi133.stdout:Adding system user ceph....done 2021-02-11T09:44:06.657 INFO:teuthology.orchestra.run.smithi133.stdout:Setting system user ceph properties....done 2021-02-11T09:44:06.658 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-02-11T09:44:06.658 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-02-11T09:44:06.658 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up radosgw (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.658 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-11T09:44:06.658 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-11T09:44:06.658 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-test (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.658 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-base (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:06.802 INFO:teuthology.orchestra.run.smithi026.stdout:Setting system user ceph properties....done 2021-02-11T09:44:06.910 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-02-11T09:44:06.962 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-02-11T09:44:07.195 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:07.217 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-02-11T09:44:07.413 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-11T09:44:07.413 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-11T09:44:07.516 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up radosgw (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:07.696 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:07.772 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-11T09:44:07.772 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-11T09:44:07.839 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-osd (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:08.193 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-test (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:08.247 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-11T09:44:08.248 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-11T09:44:08.453 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-base (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:08.540 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr-cephadm (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:08.659 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr-rook (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:08.772 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-02-11T09:44:08.777 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mds (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:08.972 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-11T09:44:08.973 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-11T09:44:09.088 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:09.245 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mon (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:09.293 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-11T09:44:09.293 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-11T09:44:09.573 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:09.637 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-11T09:44:09.637 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-11T09:44:09.708 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-osd (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:09.897 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr-k8sevents (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:09.998 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr-dashboard (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:10.124 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:10.140 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-11T09:44:10.140 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-11T09:44:10.234 INFO:teuthology.orchestra.run.smithi133.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-02-11T09:44:10.418 INFO:teuthology.orchestra.run.smithi133.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-02-11T09:44:10.419 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr-cephadm (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:10.545 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr-rook (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:10.696 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mds (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:10.907 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-11T09:44:10.907 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-11T09:44:11.206 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mon (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:11.407 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-11T09:44:11.407 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-11T09:44:11.698 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr-k8sevents (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:11.849 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph-mgr-dashboard (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:11.992 INFO:teuthology.orchestra.run.smithi026.stdout:Setting up ceph (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:44:12.119 INFO:teuthology.orchestra.run.smithi026.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-02-11T09:44:12.345 INFO:teuthology.orchestra.run.smithi026.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-02-11T09:44:12.438 INFO:teuthology.orchestra.run.smithi133.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-02-11T09:44:12.480 INFO:teuthology.orchestra.run.smithi133.stdout:ureadahead will be reprofiled on next reboot 2021-02-11T09:44:12.538 INFO:teuthology.orchestra.run.smithi133.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-02-11T09:44:12.989 INFO:teuthology.orchestra.run.smithi133.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-02-11T09:44:12.992 DEBUG:teuthology.parallel:result is None 2021-02-11T09:44:14.431 INFO:teuthology.orchestra.run.smithi026.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-02-11T09:44:14.465 INFO:teuthology.orchestra.run.smithi026.stdout:ureadahead will be reprofiled on next reboot 2021-02-11T09:44:14.515 INFO:teuthology.orchestra.run.smithi026.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-02-11T09:44:14.981 INFO:teuthology.orchestra.run.smithi026.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-02-11T09:44:14.983 DEBUG:teuthology.parallel:result is None 2021-02-11T09:44:14.984 DEBUG:teuthology.orchestra.run.smithi026:> sudo lsb_release -is 2021-02-11T09:44:15.031 INFO:teuthology.orchestra.run.smithi026.stdout:Ubuntu 2021-02-11T09:44:15.035 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T09:44:15.036 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=5e5bb33c614118d912ca83c2e5f2be67b3acc04f 2021-02-11T09:44:15.225 DEBUG:teuthology.orchestra.run.smithi026:> dpkg-query -W -f '${Version}' ceph 2021-02-11T09:44:15.249 INFO:teuthology.orchestra.run.smithi026.stdout:17.0.0-735-g5e5bb33c-1bionic 2021-02-11T09:44:15.250 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-735-g5e5bb33c-1bionic 2021-02-11T09:44:15.250 INFO:teuthology.task.install:The correct ceph version 17.0.0-735-g5e5bb33c-1bionic is installed. 2021-02-11T09:44:15.250 DEBUG:teuthology.orchestra.run.smithi133:> sudo lsb_release -is 2021-02-11T09:44:15.306 INFO:teuthology.orchestra.run.smithi133.stdout:Ubuntu 2021-02-11T09:44:15.311 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T09:44:15.311 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=5e5bb33c614118d912ca83c2e5f2be67b3acc04f 2021-02-11T09:44:15.490 DEBUG:teuthology.orchestra.run.smithi133:> dpkg-query -W -f '${Version}' ceph 2021-02-11T09:44:15.514 INFO:teuthology.orchestra.run.smithi133.stdout:17.0.0-735-g5e5bb33c-1bionic 2021-02-11T09:44:15.514 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-735-g5e5bb33c-1bionic 2021-02-11T09:44:15.514 INFO:teuthology.task.install:The correct ceph version 17.0.0-735-g5e5bb33c-1bionic is installed. 2021-02-11T09:44:15.515 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-02-11T09:44:15.515 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:15.515 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-11T09:44:15.530 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:44:15.530 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-11T09:44:15.574 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-02-11T09:44:15.575 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:15.575 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/daemon-helper 2021-02-11T09:44:15.589 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-11T09:44:15.647 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:44:15.647 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/daemon-helper 2021-02-11T09:44:15.662 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-11T09:44:15.718 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-02-11T09:44:15.718 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:15.718 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-11T09:44:15.733 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-11T09:44:15.790 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:44:15.790 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-11T09:44:15.804 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-11T09:44:15.862 INFO:teuthology.run_tasks:Running task ceph... 2021-02-11T09:44:15.946 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-02-11T09:44:15.947 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 777 /var/log/ceph 2021-02-11T09:44:15.949 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 777 /var/log/ceph 2021-02-11T09:44:15.963 INFO:tasks.ceph:Disabling ceph logrotate... 2021-02-11T09:44:15.963 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/logrotate.d/ceph 2021-02-11T09:44:16.006 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/logrotate.d/ceph 2021-02-11T09:44:16.020 INFO:tasks.ceph:Creating extra log directories... 2021-02-11T09:44:16.020 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-02-11T09:44:16.058 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-02-11T09:44:16.088 INFO:tasks.ceph:Creating ceph cluster c1... 2021-02-11T09:44:16.088 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-02-11T09:44:16.088 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874503', 'branch': 'wip-blurpletastic', 'description': 'rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'aemerson@redhat.com', 'first_in_suite': False, 'job_id': '5874503', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-blurpletastic'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f'}}, 'rgw': {'compression type': 'random', 'frontend': 'civetweb'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-blurpletastic', 'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f'}}, 'owner': 'scheduled_aemerson@teuthology', 'priority': 70, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-blurpletastic', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f', 'targets': {'smithi026.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBDwsvOghUsw0AfpZm6OwXwfU7McUxVq6l7RtzNuUCIs2rKPswTlou++DeBtTVmRqJTIvOyW9OoQsjokpItJ041v5vF6BHqilMGhMlgYOzTf38D5IDODjNCE2gWbGEQWgT9RLoBsSxSboqj4JGkGb5RlJorJkG79sce4RT+vjRaDxX3SPXlEGhut2cND0DigGU2o95eMaY+el+7ro58L3cGymLBEQBgdqAG0ylRGOWaJs2IWRQefwZOOt9UMbQNWtvXOd8CQb6B/JVaqj5OAJoRkrbQY2E5V9/oS7kc8EH8EsNWHyStaHB9lSblIVsDAapREJ+ceLCtJZ4vSlLimBp', 'smithi133.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDApmU2RgrDF6OBBSRN7KgTBoRZ18J5CDi+nwxXLLj06kaIh8EVEmOUck98z8UUzSnuJOxY3IzU7xPZwgg8CXnqTn3rdQl6qIh3PU0VReuWmSqZ07BCWiclTLDXQ0ldeDP/WUTKGHVR8FBsCEjOa1Ns7DP0txtTPVkjpRQhTW96V0oWdP9V5NcKrFpOKnkxrQANTim+ih1cMw5r9mBTQQT7IhHS1c5hwAe7JhKIVP+Ur6HhsCgltW9hfC4Wt686bNcI1HHEDA2SI7xQShPKbLq32LP59ifbRaSe5j8voANtgFnK/R2cSW/sonM55p7BRgZlNow4qATJS09nrT3XbdyF'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2021-02-11_09:24:13', 'tube': 'smithi', 'user': 'aemerson', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.29888'} 2021-02-11T09:44:16.089 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-02-11T09:44:16.114 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-02-11T09:44:16.126 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/run/ceph 2021-02-11T09:44:16.162 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -d -m0777 -- /var/run/ceph 2021-02-11T09:44:16.182 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:16.182 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/scratch_devs of=/dev/stdout 2021-02-11T09:44:16.223 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-11T09:44:16.223 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-11T09:44:16.271 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-02-11T09:44:16.271 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T09:44:16.271 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 6h/6d Inode: 341 Links: 1 2021-02-11T09:44:16.271 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T09:44:16.271 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2021-02-11 09:41:04.630333015 +0000 2021-02-11T09:44:16.271 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2021-02-11 09:41:04.522333017 +0000 2021-02-11T09:44:16.271 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2021-02-11 09:41:04.522333017 +0000 2021-02-11T09:44:16.272 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: - 2021-02-11T09:44:16.281 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2021-02-11T09:44:16.281 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2021-02-11T09:44:16.281 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000415287 s, 1.2 MB/s 2021-02-11T09:44:16.286 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-11T09:44:16.335 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-02-11T09:44:16.335 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T09:44:16.335 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 6h/6d Inode: 348 Links: 1 2021-02-11T09:44:16.335 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T09:44:16.335 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2021-02-11 09:41:04.878333009 +0000 2021-02-11T09:44:16.336 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2021-02-11 09:41:04.778333012 +0000 2021-02-11T09:44:16.336 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2021-02-11 09:41:04.778333012 +0000 2021-02-11T09:44:16.336 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: - 2021-02-11T09:44:16.345 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2021-02-11T09:44:16.345 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2021-02-11T09:44:16.345 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000388847 s, 1.3 MB/s 2021-02-11T09:44:16.349 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-11T09:44:16.399 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-02-11T09:44:16.399 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T09:44:16.399 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 6h/6d Inode: 354 Links: 1 2021-02-11T09:44:16.399 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T09:44:16.399 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2021-02-11 09:41:05.158333003 +0000 2021-02-11T09:44:16.400 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2021-02-11 09:41:05.046333006 +0000 2021-02-11T09:44:16.400 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2021-02-11 09:41:05.046333006 +0000 2021-02-11T09:44:16.400 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: - 2021-02-11T09:44:16.409 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2021-02-11T09:44:16.409 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2021-02-11T09:44:16.409 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000392223 s, 1.3 MB/s 2021-02-11T09:44:16.414 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-11T09:44:16.467 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-02-11T09:44:16.467 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T09:44:16.467 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 6h/6d Inode: 360 Links: 1 2021-02-11T09:44:16.467 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T09:44:16.468 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2021-02-11 09:41:05.390332998 +0000 2021-02-11T09:44:16.468 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2021-02-11 09:41:05.302333000 +0000 2021-02-11T09:44:16.468 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2021-02-11 09:41:05.302333000 +0000 2021-02-11T09:44:16.468 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: - 2021-02-11T09:44:16.476 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2021-02-11T09:44:16.476 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2021-02-11T09:44:16.476 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000372537 s, 1.4 MB/s 2021-02-11T09:44:16.480 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T09:44:16.480 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-11T09:44:16.480 INFO:tasks.ceph:Generating config... 2021-02-11T09:44:16.482 INFO:tasks.ceph:[client] debug rgw = 20 2021-02-11T09:44:16.482 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-02-11T09:44:16.482 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-02-11T09:44:16.482 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T09:44:16.482 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-02-11T09:44:16.482 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-02-11T09:44:16.482 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-02-11T09:44:16.482 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-02-11T09:44:16.482 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-02-11T09:44:16.482 INFO:tasks.ceph:[client] setgroup = ceph 2021-02-11T09:44:16.482 INFO:tasks.ceph:[client] setuser = ceph 2021-02-11T09:44:16.483 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2021-02-11T09:44:16.483 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-02-11T09:44:16.483 INFO:tasks.ceph:[mgr] debug ms = 1 2021-02-11T09:44:16.483 INFO:tasks.ceph:[mon] debug mon = 20 2021-02-11T09:44:16.483 INFO:tasks.ceph:[mon] debug ms = 1 2021-02-11T09:44:16.483 INFO:tasks.ceph:[mon] debug paxos = 20 2021-02-11T09:44:16.483 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2021-02-11T09:44:16.483 INFO:tasks.ceph:[osd] debug filestore = 20 2021-02-11T09:44:16.483 INFO:tasks.ceph:[osd] debug journal = 20 2021-02-11T09:44:16.483 INFO:tasks.ceph:[osd] debug ms = 20 2021-02-11T09:44:16.483 INFO:tasks.ceph:[osd] debug osd = 25 2021-02-11T09:44:16.483 INFO:tasks.ceph:[osd] osd fast shutdown = False 2021-02-11T09:44:16.483 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-02-11T09:44:16.483 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-02-11T09:44:16.484 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-02-11T09:44:16.484 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-02-11T09:44:16.484 INFO:tasks.ceph:Setting up c1.mon.a... 2021-02-11T09:44:16.484 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2021-02-11T09:44:16.552 INFO:teuthology.orchestra.run.smithi026.stdout:creating /etc/ceph/c1.keyring 2021-02-11T09:44:16.554 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2021-02-11T09:44:16.630 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-02-11T09:44:16.686 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.26')] 2021-02-11T09:44:16.686 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.26', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-02-11T09:44:16.687 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:16.687 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-02-11T09:44:16.734 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.26 --print /home/ubuntu/cephtest/c1.monmap 2021-02-11T09:44:16.807 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:44:16.807 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:44:16.807 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2021-02-11T09:44:16.807 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: generated fsid 4594769f-0932-4e4a-a0a6-4220b143584b 2021-02-11T09:44:16.807 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 0 2021-02-11T09:44:16.807 INFO:teuthology.orchestra.run.smithi026.stdout:fsid 4594769f-0932-4e4a-a0a6-4220b143584b 2021-02-11T09:44:16.807 INFO:teuthology.orchestra.run.smithi026.stdout:last_changed 2021-02-11T09:44:16.809994+0000 2021-02-11T09:44:16.808 INFO:teuthology.orchestra.run.smithi026.stdout:created 2021-02-11T09:44:16.809994+0000 2021-02-11T09:44:16.808 INFO:teuthology.orchestra.run.smithi026.stdout:min_mon_release 0 (unknown) 2021-02-11T09:44:16.808 INFO:teuthology.orchestra.run.smithi026.stdout:election_strategy: 1 2021-02-11T09:44:16.808 INFO:teuthology.orchestra.run.smithi026.stdout:0: [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon.a 2021-02-11T09:44:16.808 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2021-02-11T09:44:16.809 DEBUG:teuthology.orchestra.run.smithi026:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-02-11T09:44:16.858 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 4594769f-0932-4e4a-a0a6-4220b143584b... 2021-02-11T09:44:16.859 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-02-11T09:44:16.902 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-02-11T09:44:16.932 INFO:teuthology.orchestra.run.smithi026.stdout:[global] 2021-02-11T09:44:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: chdir = "" 2021-02-11T09:44:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-11T09:44:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: auth supported = cephx 2021-02-11T09:44:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: filestore xattr use omap = true 2021-02-11T09:44:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: mon clock drift allowed = 1.000 2021-02-11T09:44:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: osd crush chooseleaf type = 0 2021-02-11T09:44:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: auth debug = true 2021-02-11T09:44:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on old message = true 2021-02-11T09:44:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on bug = true 2021-02-11T09:44:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-11T09:44:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: mon pg warn max object skew = 0 2021-02-11T09:44:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: # disable pg_autoscaler by default for new pools 2021-02-11T09:44:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-11T09:44:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default size = 2 2021-02-11T09:44:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow primary affinity = true 2021-02-11T09:44:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow pg remap = true 2021-02-11T09:44:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on legacy crush tunables = false 2021-02-11T09:44:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on crush straw calc version zero = false 2021-02-11T09:44:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on no sortbitwise = false 2021-02-11T09:44:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on osd down out interval zero = false 2021-02-11T09:44:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on too few osds = false 2021-02-11T09:44:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-11T09:44:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-11T09:44:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: mon_allow_pool_size_one = true 2021-02-11T09:44:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-11T09:44:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: osd default data pool replay window = 5 2021-02-11T09:44:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: mon allow pool delete = true 2021-02-11T09:44:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: mon cluster log file level = debug 2021-02-11T09:44:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: debug asserts on shutdown = true 2021-02-11T09:44:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: mon health detail to clog = false 2021-02-11T09:44:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: mon host = 172.21.15.26 2021-02-11T09:44:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: osd heartbeat grace = 40 2021-02-11T09:44:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: fsid = 4594769f-0932-4e4a-a0a6-4220b143584b 2021-02-11T09:44:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.935 INFO:teuthology.orchestra.run.smithi026.stdout:[osd] 2021-02-11T09:44:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: osd journal size = 100 2021-02-11T09:44:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub load threshold = 5.0 2021-02-11T09:44:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub max interval = 600 2021-02-11T09:44:16.936 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.936 INFO:teuthology.orchestra.run.smithi026.stdout: osd recover clone overlap = true 2021-02-11T09:44:16.936 INFO:teuthology.orchestra.run.smithi026.stdout: osd recovery max chunk = 1048576 2021-02-11T09:44:16.936 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.936 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug shutdown = true 2021-02-11T09:44:16.936 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug op order = true 2021-02-11T09:44:16.936 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify stray on activate = true 2021-02-11T09:44:16.936 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.936 INFO:teuthology.orchestra.run.smithi026.stdout: osd open classes on start = true 2021-02-11T09:44:16.936 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug pg log writeout = true 2021-02-11T09:44:16.936 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.936 INFO:teuthology.orchestra.run.smithi026.stdout: osd deep scrub update digest min age = 30 2021-02-11T09:44:16.936 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.936 INFO:teuthology.orchestra.run.smithi026.stdout: osd map max advance = 10 2021-02-11T09:44:16.937 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.937 INFO:teuthology.orchestra.run.smithi026.stdout: journal zero on create = true 2021-02-11T09:44:16.937 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.937 INFO:teuthology.orchestra.run.smithi026.stdout: filestore ondisk finisher threads = 3 2021-02-11T09:44:16.937 INFO:teuthology.orchestra.run.smithi026.stdout: filestore apply finisher threads = 3 2021-02-11T09:44:16.937 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.937 INFO:teuthology.orchestra.run.smithi026.stdout: bdev debug aio = true 2021-02-11T09:44:16.937 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug misdirected ops = true 2021-02-11T09:44:16.937 INFO:teuthology.orchestra.run.smithi026.stdout: debug filestore = 20 2021-02-11T09:44:16.937 INFO:teuthology.orchestra.run.smithi026.stdout: debug journal = 20 2021-02-11T09:44:16.937 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 20 2021-02-11T09:44:16.937 INFO:teuthology.orchestra.run.smithi026.stdout: debug osd = 25 2021-02-11T09:44:16.937 INFO:teuthology.orchestra.run.smithi026.stdout: osd fast shutdown = False 2021-02-11T09:44:16.938 INFO:teuthology.orchestra.run.smithi026.stdout: rocksdb delete range threshold = 0 2021-02-11T09:44:16.938 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.938 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr] 2021-02-11T09:44:16.938 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2021-02-11T09:44:16.938 INFO:teuthology.orchestra.run.smithi026.stdout: debug mgr = 20 2021-02-11T09:44:16.938 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2021-02-11T09:44:16.938 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2021-02-11T09:44:16.938 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2021-02-11T09:44:16.938 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2021-02-11T09:44:16.938 INFO:teuthology.orchestra.run.smithi026.stdout: mgr/telemetry/nag = false 2021-02-11T09:44:16.938 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.938 INFO:teuthology.orchestra.run.smithi026.stdout:[mon] 2021-02-11T09:44:16.938 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2021-02-11T09:44:16.938 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2021-02-11T09:44:16.939 INFO:teuthology.orchestra.run.smithi026.stdout: debug paxos = 20 2021-02-11T09:44:16.939 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2021-02-11T09:44:16.939 INFO:teuthology.orchestra.run.smithi026.stdout: mon data avail warn = 5 2021-02-11T09:44:16.939 INFO:teuthology.orchestra.run.smithi026.stdout: mon mgr mkfs grace = 240 2021-02-11T09:44:16.939 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2021-02-11T09:44:16.939 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd reporter subtree level = osd 2021-02-11T09:44:16.939 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd prime pg temp = true 2021-02-11T09:44:16.939 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2021-02-11T09:44:16.939 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd crush smoke test = False 2021-02-11T09:44:16.939 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:16.939 INFO:teuthology.orchestra.run.smithi026.stdout:[client] 2021-02-11T09:44:16.939 INFO:teuthology.orchestra.run.smithi026.stdout: rgw cache enabled = true 2021-02-11T09:44:16.939 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable ops log = true 2021-02-11T09:44:16.939 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable usage log = true 2021-02-11T09:44:16.940 INFO:teuthology.orchestra.run.smithi026.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-11T09:44:16.940 INFO:teuthology.orchestra.run.smithi026.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-11T09:44:16.940 INFO:teuthology.orchestra.run.smithi026.stdout: debug rgw = 20 2021-02-11T09:44:16.940 INFO:teuthology.orchestra.run.smithi026.stdout: rgw crypt require ssl = False 2021-02-11T09:44:16.940 INFO:teuthology.orchestra.run.smithi026.stdout: rgw crypt s3 kms backend = testing 2021-02-11T09:44:16.940 INFO:teuthology.orchestra.run.smithi026.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T09:44:16.940 INFO:teuthology.orchestra.run.smithi026.stdout: rgw curl low speed time = 300 2021-02-11T09:44:16.940 INFO:teuthology.orchestra.run.smithi026.stdout: rgw data log num shards = 4 2021-02-11T09:44:16.940 INFO:teuthology.orchestra.run.smithi026.stdout: rgw md log max shards = 4 2021-02-11T09:44:16.940 INFO:teuthology.orchestra.run.smithi026.stdout: rgw sync log trim interval = 0 2021-02-11T09:44:16.940 INFO:teuthology.orchestra.run.smithi026.stdout: rgw sync obj etag verify = True 2021-02-11T09:44:16.940 INFO:teuthology.orchestra.run.smithi026.stdout: setgroup = ceph 2021-02-11T09:44:16.940 INFO:teuthology.orchestra.run.smithi026.stdout: setuser = ceph 2021-02-11T09:44:16.940 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.a] 2021-02-11T09:44:16.941 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.0] 2021-02-11T09:44:16.941 INFO:teuthology.orchestra.run.smithi026.stdout: osd_max_omap_entries_per_request = 10 2021-02-11T09:44:16.941 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.1] 2021-02-11T09:44:16.941 INFO:teuthology.orchestra.run.smithi026.stdout: osd_max_omap_entries_per_request = 1000 2021-02-11T09:44:16.941 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.2] 2021-02-11T09:44:16.941 INFO:teuthology.orchestra.run.smithi026.stdout: osd_max_omap_entries_per_request = 10000 2021-02-11T09:44:16.942 INFO:teuthology.orchestra.run.smithi133.stdout:[global] 2021-02-11T09:44:16.942 INFO:teuthology.orchestra.run.smithi133.stdout: chdir = "" 2021-02-11T09:44:16.942 INFO:teuthology.orchestra.run.smithi133.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-11T09:44:16.942 INFO:teuthology.orchestra.run.smithi133.stdout: auth supported = cephx 2021-02-11T09:44:16.942 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.942 INFO:teuthology.orchestra.run.smithi133.stdout: filestore xattr use omap = true 2021-02-11T09:44:16.942 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.942 INFO:teuthology.orchestra.run.smithi133.stdout: mon clock drift allowed = 1.000 2021-02-11T09:44:16.943 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.943 INFO:teuthology.orchestra.run.smithi133.stdout: osd crush chooseleaf type = 0 2021-02-11T09:44:16.943 INFO:teuthology.orchestra.run.smithi133.stdout: auth debug = true 2021-02-11T09:44:16.943 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.943 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on old message = true 2021-02-11T09:44:16.943 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on bug = true 2021-02-11T09:44:16.943 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.943 INFO:teuthology.orchestra.run.smithi133.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-11T09:44:16.943 INFO:teuthology.orchestra.run.smithi133.stdout: mon pg warn max object skew = 0 2021-02-11T09:44:16.943 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.943 INFO:teuthology.orchestra.run.smithi133.stdout: # disable pg_autoscaler by default for new pools 2021-02-11T09:44:16.943 INFO:teuthology.orchestra.run.smithi133.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-11T09:44:16.943 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.943 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default size = 2 2021-02-11T09:44:16.944 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.944 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow primary affinity = true 2021-02-11T09:44:16.944 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow pg remap = true 2021-02-11T09:44:16.944 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on legacy crush tunables = false 2021-02-11T09:44:16.944 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on crush straw calc version zero = false 2021-02-11T09:44:16.944 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on no sortbitwise = false 2021-02-11T09:44:16.944 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on osd down out interval zero = false 2021-02-11T09:44:16.944 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on too few osds = false 2021-02-11T09:44:16.944 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-11T09:44:16.944 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-11T09:44:16.944 INFO:teuthology.orchestra.run.smithi133.stdout: mon_allow_pool_size_one = true 2021-02-11T09:44:16.944 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.944 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-11T09:44:16.944 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.945 INFO:teuthology.orchestra.run.smithi133.stdout: osd default data pool replay window = 5 2021-02-11T09:44:16.945 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.945 INFO:teuthology.orchestra.run.smithi133.stdout: mon allow pool delete = true 2021-02-11T09:44:16.945 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.945 INFO:teuthology.orchestra.run.smithi133.stdout: mon cluster log file level = debug 2021-02-11T09:44:16.945 INFO:teuthology.orchestra.run.smithi133.stdout: debug asserts on shutdown = true 2021-02-11T09:44:16.945 INFO:teuthology.orchestra.run.smithi133.stdout: mon health detail to clog = false 2021-02-11T09:44:16.945 INFO:teuthology.orchestra.run.smithi133.stdout: mon host = 172.21.15.26 2021-02-11T09:44:16.945 INFO:teuthology.orchestra.run.smithi133.stdout: osd heartbeat grace = 40 2021-02-11T09:44:16.945 INFO:teuthology.orchestra.run.smithi133.stdout: fsid = 4594769f-0932-4e4a-a0a6-4220b143584b 2021-02-11T09:44:16.945 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.945 INFO:teuthology.orchestra.run.smithi133.stdout:[osd] 2021-02-11T09:44:16.945 INFO:teuthology.orchestra.run.smithi133.stdout: osd journal size = 100 2021-02-11T09:44:16.945 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.946 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub load threshold = 5.0 2021-02-11T09:44:16.946 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub max interval = 600 2021-02-11T09:44:16.946 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.946 INFO:teuthology.orchestra.run.smithi133.stdout: osd recover clone overlap = true 2021-02-11T09:44:16.946 INFO:teuthology.orchestra.run.smithi133.stdout: osd recovery max chunk = 1048576 2021-02-11T09:44:16.946 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.946 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug shutdown = true 2021-02-11T09:44:16.946 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug op order = true 2021-02-11T09:44:16.946 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug verify stray on activate = true 2021-02-11T09:44:16.946 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.946 INFO:teuthology.orchestra.run.smithi133.stdout: osd open classes on start = true 2021-02-11T09:44:16.946 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug pg log writeout = true 2021-02-11T09:44:16.947 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.947 INFO:teuthology.orchestra.run.smithi133.stdout: osd deep scrub update digest min age = 30 2021-02-11T09:44:16.947 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.947 INFO:teuthology.orchestra.run.smithi133.stdout: osd map max advance = 10 2021-02-11T09:44:16.947 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.947 INFO:teuthology.orchestra.run.smithi133.stdout: journal zero on create = true 2021-02-11T09:44:16.947 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.947 INFO:teuthology.orchestra.run.smithi133.stdout: filestore ondisk finisher threads = 3 2021-02-11T09:44:16.947 INFO:teuthology.orchestra.run.smithi133.stdout: filestore apply finisher threads = 3 2021-02-11T09:44:16.947 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.947 INFO:teuthology.orchestra.run.smithi133.stdout: bdev debug aio = true 2021-02-11T09:44:16.947 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug misdirected ops = true 2021-02-11T09:44:16.947 INFO:teuthology.orchestra.run.smithi133.stdout: debug filestore = 20 2021-02-11T09:44:16.947 INFO:teuthology.orchestra.run.smithi133.stdout: debug journal = 20 2021-02-11T09:44:16.948 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 20 2021-02-11T09:44:16.948 INFO:teuthology.orchestra.run.smithi133.stdout: debug osd = 25 2021-02-11T09:44:16.948 INFO:teuthology.orchestra.run.smithi133.stdout: osd fast shutdown = False 2021-02-11T09:44:16.948 INFO:teuthology.orchestra.run.smithi133.stdout: rocksdb delete range threshold = 0 2021-02-11T09:44:16.948 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.948 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr] 2021-02-11T09:44:16.948 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2021-02-11T09:44:16.948 INFO:teuthology.orchestra.run.smithi133.stdout: debug mgr = 20 2021-02-11T09:44:16.948 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2021-02-11T09:44:16.948 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2021-02-11T09:44:16.948 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2021-02-11T09:44:16.948 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2021-02-11T09:44:16.948 INFO:teuthology.orchestra.run.smithi133.stdout: mgr/telemetry/nag = false 2021-02-11T09:44:16.948 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.949 INFO:teuthology.orchestra.run.smithi133.stdout:[mon] 2021-02-11T09:44:16.949 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2021-02-11T09:44:16.949 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2021-02-11T09:44:16.949 INFO:teuthology.orchestra.run.smithi133.stdout: debug paxos = 20 2021-02-11T09:44:16.949 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2021-02-11T09:44:16.949 INFO:teuthology.orchestra.run.smithi133.stdout: mon data avail warn = 5 2021-02-11T09:44:16.949 INFO:teuthology.orchestra.run.smithi133.stdout: mon mgr mkfs grace = 240 2021-02-11T09:44:16.949 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2021-02-11T09:44:16.949 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd reporter subtree level = osd 2021-02-11T09:44:16.949 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd prime pg temp = true 2021-02-11T09:44:16.949 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2021-02-11T09:44:16.949 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd crush smoke test = False 2021-02-11T09:44:16.949 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:16.949 INFO:teuthology.orchestra.run.smithi133.stdout:[client] 2021-02-11T09:44:16.950 INFO:teuthology.orchestra.run.smithi133.stdout: rgw cache enabled = true 2021-02-11T09:44:16.950 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable ops log = true 2021-02-11T09:44:16.950 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable usage log = true 2021-02-11T09:44:16.950 INFO:teuthology.orchestra.run.smithi133.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-11T09:44:16.950 INFO:teuthology.orchestra.run.smithi133.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-11T09:44:16.950 INFO:teuthology.orchestra.run.smithi133.stdout: debug rgw = 20 2021-02-11T09:44:16.950 INFO:teuthology.orchestra.run.smithi133.stdout: rgw crypt require ssl = False 2021-02-11T09:44:16.950 INFO:teuthology.orchestra.run.smithi133.stdout: rgw crypt s3 kms backend = testing 2021-02-11T09:44:16.950 INFO:teuthology.orchestra.run.smithi133.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T09:44:16.950 INFO:teuthology.orchestra.run.smithi133.stdout: rgw curl low speed time = 300 2021-02-11T09:44:16.950 INFO:teuthology.orchestra.run.smithi133.stdout: rgw data log num shards = 4 2021-02-11T09:44:16.950 INFO:teuthology.orchestra.run.smithi133.stdout: rgw md log max shards = 4 2021-02-11T09:44:16.950 INFO:teuthology.orchestra.run.smithi133.stdout: rgw sync log trim interval = 0 2021-02-11T09:44:16.951 INFO:teuthology.orchestra.run.smithi133.stdout: rgw sync obj etag verify = True 2021-02-11T09:44:16.951 INFO:teuthology.orchestra.run.smithi133.stdout: setgroup = ceph 2021-02-11T09:44:16.951 INFO:teuthology.orchestra.run.smithi133.stdout: setuser = ceph 2021-02-11T09:44:16.951 INFO:teuthology.orchestra.run.smithi133.stdout:[mon.a] 2021-02-11T09:44:16.951 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.0] 2021-02-11T09:44:16.951 INFO:teuthology.orchestra.run.smithi133.stdout: osd_max_omap_entries_per_request = 10 2021-02-11T09:44:16.951 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.1] 2021-02-11T09:44:16.951 INFO:teuthology.orchestra.run.smithi133.stdout: osd_max_omap_entries_per_request = 1000 2021-02-11T09:44:16.951 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.2] 2021-02-11T09:44:16.951 INFO:teuthology.orchestra.run.smithi133.stdout: osd_max_omap_entries_per_request = 10000 2021-02-11T09:44:16.952 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2021-02-11T09:44:16.952 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2021-02-11T09:44:17.030 INFO:tasks.ceph:Copying monmap to all nodes... 2021-02-11T09:44:17.030 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:17.030 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2021-02-11T09:44:17.079 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:17.079 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2021-02-11T09:44:17.126 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi026.front.sepia.ceph.com 2021-02-11T09:44:17.126 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:17.126 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/c1.keyring 2021-02-11T09:44:17.126 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-02-11T09:44:17.193 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:17.194 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-02-11T09:44:17.242 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi133.front.sepia.ceph.com 2021-02-11T09:44:17.242 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:44:17.242 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/ceph/c1.keyring 2021-02-11T09:44:17.242 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-02-11T09:44:17.266 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:44:17.266 DEBUG:teuthology.orchestra.run.smithi133:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-02-11T09:44:17.314 INFO:tasks.ceph:Setting up mon nodes... 2021-02-11T09:44:17.314 INFO:tasks.ceph:Setting up mgr nodes... 2021-02-11T09:44:17.314 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2021-02-11T09:44:17.357 INFO:teuthology.orchestra.run.smithi026.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2021-02-11T09:44:17.359 INFO:tasks.ceph:Setting up mds nodes... 2021-02-11T09:44:17.359 INFO:tasks.ceph_client:Setting up client nodes... 2021-02-11T09:44:17.360 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2021-02-11T09:44:17.433 INFO:teuthology.orchestra.run.smithi026.stdout:creating /etc/ceph/c1.client.0.keyring 2021-02-11T09:44:17.446 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2021-02-11T09:44:17.521 INFO:teuthology.orchestra.run.smithi026.stdout:creating /etc/ceph/c1.client.1.keyring 2021-02-11T09:44:17.533 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-02-11T09:44:17.533 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-11T09:44:17.533 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2021-02-11T09:44:17.591 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T09:44:17.591 INFO:tasks.ceph:role: c1.osd.0 2021-02-11T09:44:17.591 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi026.front.sepia.ceph.com 2021-02-11T09:44:17.592 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-02-11T09:44:18.103 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T09:44:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T09:44:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T09:44:18.104 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T09:44:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2021-02-11T09:44:18.104 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T09:44:18.104 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T09:44:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T09:44:18.104 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T09:44:18.105 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2021-02-11T09:44:18.105 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2021-02-11T09:44:18.181 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2021-02-11T09:44:18.191 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T09:44:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T09:44:18.191 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2021-02-11T09:44:18.247 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T09:44:18.247 INFO:tasks.ceph:role: c1.osd.1 2021-02-11T09:44:18.247 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi026.front.sepia.ceph.com 2021-02-11T09:44:18.248 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-02-11T09:44:18.767 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T09:44:18.768 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T09:44:18.768 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T09:44:18.768 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T09:44:18.768 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2021-02-11T09:44:18.768 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T09:44:18.768 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T09:44:18.768 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T09:44:18.768 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T09:44:18.769 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2021-02-11T09:44:18.769 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2021-02-11T09:44:18.833 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2021-02-11T09:44:18.844 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T09:44:18.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T09:44:18.845 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2021-02-11T09:44:18.903 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T09:44:18.903 INFO:tasks.ceph:role: c1.osd.2 2021-02-11T09:44:18.903 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi026.front.sepia.ceph.com 2021-02-11T09:44:18.903 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-02-11T09:44:19.404 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T09:44:19.404 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T09:44:19.404 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T09:44:19.404 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T09:44:19.404 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2021-02-11T09:44:19.404 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T09:44:19.404 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T09:44:19.404 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T09:44:19.404 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T09:44:19.405 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2021-02-11T09:44:19.405 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2021-02-11T09:44:19.452 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2021-02-11T09:44:19.462 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T09:44:19.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T09:44:19.463 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2021-02-11T09:44:19.554 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:44:19.553+0000 7f3082453080 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2021-02-11T09:44:19.555 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:44:19.553+0000 7f3082453080 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2021-02-11T09:44:19.556 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:44:19.553+0000 7f3082453080 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-02-11T09:44:19.556 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:44:19.553+0000 7f3082453080 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-02-11T09:44:19.556 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:44:19.553+0000 7f3082453080 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-02-11T09:44:19.559 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:44:19.557+0000 7f3082453080 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2021-02-11T09:44:21.744 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2021-02-11T09:44:21.760 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2021-02-11T09:44:21.843 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:44:21.841+0000 7f144e83e080 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2021-02-11T09:44:21.843 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:44:21.841+0000 7f144e83e080 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2021-02-11T09:44:21.843 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:44:21.841+0000 7f144e83e080 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-02-11T09:44:21.843 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:44:21.841+0000 7f144e83e080 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-02-11T09:44:21.843 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:44:21.841+0000 7f144e83e080 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-02-11T09:44:21.844 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:44:21.841+0000 7f144e83e080 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2021-02-11T09:44:24.025 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2021-02-11T09:44:24.083 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2021-02-11T09:44:24.165 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:44:24.165+0000 7ff9a05d9080 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2021-02-11T09:44:24.165 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:44:24.165+0000 7ff9a05d9080 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2021-02-11T09:44:24.166 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:44:24.165+0000 7ff9a05d9080 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-02-11T09:44:24.166 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:44:24.165+0000 7ff9a05d9080 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-02-11T09:44:24.166 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:44:24.165+0000 7ff9a05d9080 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-02-11T09:44:24.167 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:44:24.165+0000 7ff9a05d9080 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2021-02-11T09:44:26.337 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2021-02-11T09:44:26.396 INFO:tasks.ceph:Reading keys from all nodes... 2021-02-11T09:44:26.396 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:26.396 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2021-02-11T09:44:26.451 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:26.451 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2021-02-11T09:44:26.507 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:26.507 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2021-02-11T09:44:26.561 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:26.561 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2021-02-11T09:44:26.618 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:26.619 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2021-02-11T09:44:26.665 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:26.666 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2021-02-11T09:44:26.714 INFO:tasks.ceph:Adding keys to all mons... 2021-02-11T09:44:26.714 DEBUG:teuthology.orchestra.run.smithi026:> sudo tee -a /etc/ceph/c1.keyring 2021-02-11T09:44:26.772 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr.x] 2021-02-11T09:44:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQBx/CRg8BxzFRAAaDXh7e1kEgSHeAhLC/oULw== 2021-02-11T09:44:26.772 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.0] 2021-02-11T09:44:26.772 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQBz/CRgJpU3IRAAFCoc/+ifqlreUsmzfEEQHA== 2021-02-11T09:44:26.772 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.1] 2021-02-11T09:44:26.773 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQB1/CRgfVBkMhAAOaIWa2SV+IKjGp/wxj5TEw== 2021-02-11T09:44:26.773 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.2] 2021-02-11T09:44:26.773 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQB4/CRgUzcHChAAOKmeyGf1udHJ2fhwwFfH+A== 2021-02-11T09:44:26.773 INFO:teuthology.orchestra.run.smithi026.stdout:[client.0] 2021-02-11T09:44:26.773 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQBx/CRg34//GRAA9iU8yZRtxdcGMJ6kGSfZpw== 2021-02-11T09:44:26.773 INFO:teuthology.orchestra.run.smithi026.stdout:[client.1] 2021-02-11T09:44:26.773 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQBx/CRg9zM8HxAAsDfi/6gSK4l7TqRzY3bTcw== 2021-02-11T09:44:26.774 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-11T09:44:26.858 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T09:44:26.943 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T09:44:27.027 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T09:44:27.113 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-11T09:44:27.202 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-11T09:44:27.242 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-02-11T09:44:27.243 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2021-02-11T09:44:27.299 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2021-02-11T09:44:27.395 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2021-02-11T09:44:27.409 DEBUG:teuthology.orchestra.run.smithi026:> rm -- /home/ubuntu/cephtest/c1.monmap 2021-02-11T09:44:27.458 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2021-02-11T09:44:27.458 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2021-02-11T09:44:27.458 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2021-02-11T09:44:27.502 INFO:tasks.ceph.c1.mon.a:Started 2021-02-11T09:44:27.502 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2021-02-11T09:44:27.502 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2021-02-11T09:44:27.502 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2021-02-11T09:44:27.504 INFO:tasks.ceph.c1.mgr.x:Started 2021-02-11T09:44:27.505 INFO:tasks.ceph:Setting crush tunables to default 2021-02-11T09:44:27.505 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster c1 osd crush tunables default 2021-02-11T09:44:27.915 INFO:teuthology.orchestra.run.smithi026.stderr:adjusted tunables profile to default 2021-02-11T09:44:27.928 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2021-02-11T09:44:27.928 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:27.928 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2021-02-11T09:44:27.942 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:27.943 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2021-02-11T09:44:27.996 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:27.996 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2021-02-11T09:44:28.050 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster c1 osd new 73968cb7-66da-4bd6-9b13-f436aa3f1654 0 2021-02-11T09:44:28.325 INFO:teuthology.orchestra.run.smithi026.stdout:0 2021-02-11T09:44:28.340 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster c1 osd new c3548235-9f89-49fd-a481-722c3fbad94c 1 2021-02-11T09:44:28.558 INFO:teuthology.orchestra.run.smithi026.stdout:1 2021-02-11T09:44:28.572 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster c1 osd new 8048fbe2-5786-4d39-83a4-853777450c19 2 2021-02-11T09:44:28.791 INFO:teuthology.orchestra.run.smithi026.stdout:2 2021-02-11T09:44:28.804 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2021-02-11T09:44:28.804 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2021-02-11T09:44:28.806 INFO:tasks.ceph.c1.osd.0:Started 2021-02-11T09:44:28.806 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2021-02-11T09:44:28.806 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2021-02-11T09:44:28.808 INFO:tasks.ceph.c1.osd.1:Started 2021-02-11T09:44:28.808 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2021-02-11T09:44:28.808 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2021-02-11T09:44:28.810 INFO:tasks.ceph.c1.osd.2:Started 2021-02-11T09:44:28.810 INFO:tasks.ceph:Waiting for OSDs to come up 2021-02-11T09:44:29.113 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-02-11T09:44:29.303 INFO:teuthology.misc.health.smithi026.stdout: 2021-02-11T09:44:29.303 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":5,"fsid":"4594769f-0932-4e4a-a0a6-4220b143584b","created":"2021-02-11T09:44:27.643251+0000","modified":"2021-02-11T09:44:28.791512+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"73968cb7-66da-4bd6-9b13-f436aa3f1654","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3548235-9f89-49fd-a481-722c3fbad94c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8048fbe2-5786-4d39-83a4-853777450c19","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:44:29.321 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-02-11T09:44:29.430 INFO:tasks.ceph.c1.osd.0.smithi026.stderr:2021-02-11T09:44:29.429+0000 7f7d0a3be080 -1 Falling back to public interface 2021-02-11T09:44:29.446 INFO:tasks.ceph.c1.osd.1.smithi026.stderr:2021-02-11T09:44:29.445+0000 7fa6e4c4b080 -1 Falling back to public interface 2021-02-11T09:44:29.454 INFO:tasks.ceph.c1.osd.2.smithi026.stderr:2021-02-11T09:44:29.453+0000 7f88c8144080 -1 Falling back to public interface 2021-02-11T09:44:32.142 INFO:tasks.ceph.c1.osd.1.smithi026.stderr:2021-02-11T09:44:32.141+0000 7fa6e4c4b080 -1 osd.1 0 log_to_monitors {default=true} 2021-02-11T09:44:47.946 INFO:tasks.ceph.c1.osd.0.smithi026.stderr:2021-02-11T09:44:32.361+0000 7f7d0a3be080 -1 osd.0 0 log_to_monitors {default=true} 2021-02-11T09:44:47.946 INFO:tasks.ceph.c1.osd.0.smithi026.stderr:2021-02-11T09:44:34.225+0000 7f7d01887700 -1 osd.0 0 waiting for initial osdmap 2021-02-11T09:44:47.946 INFO:tasks.ceph.c1.osd.2.smithi026.stderr:2021-02-11T09:44:32.465+0000 7f88c8144080 -1 osd.2 0 log_to_monitors {default=true} 2021-02-11T09:44:47.946 INFO:tasks.ceph.c1.osd.2.smithi026.stderr:2021-02-11T09:44:34.225+0000 7f88bf60d700 -1 osd.2 0 waiting for initial osdmap 2021-02-11T09:44:47.946 INFO:tasks.ceph.c1.osd.1.smithi026.stderr:2021-02-11T09:44:33.217+0000 7fa6dc114700 -1 osd.1 0 waiting for initial osdmap 2021-02-11T09:44:48.246 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-02-11T09:44:48.541 INFO:teuthology.misc.health.smithi026.stdout: 2021-02-11T09:44:48.541 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":12,"fsid":"4594769f-0932-4e4a-a0a6-4220b143584b","created":"2021-02-11T09:44:27.643251+0000","modified":"2021-02-11T09:44:38.236217+0000","last_up_change":"2021-02-11T09:44:35.223423+0000","last_in_change":"2021-02-11T09:44:35.223423+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:44:35.233562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"73968cb7-66da-4bd6-9b13-f436aa3f1654","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":14584},{"type":"v1","addr":"172.21.15.26:6801","nonce":14584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":14584},{"type":"v1","addr":"172.21.15.26:6803","nonce":14584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":14584},{"type":"v1","addr":"172.21.15.26:6807","nonce":14584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":14584},{"type":"v1","addr":"172.21.15.26:6805","nonce":14584}]},"public_addr":"172.21.15.26:6801/14584","cluster_addr":"172.21.15.26:6803/14584","heartbeat_back_addr":"172.21.15.26:6807/14584","heartbeat_front_addr":"172.21.15.26:6805/14584","state":["exists","up"]},{"osd":1,"uuid":"c3548235-9f89-49fd-a481-722c3fbad94c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":14582},{"type":"v1","addr":"172.21.15.26:6809","nonce":14582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":14582},{"type":"v1","addr":"172.21.15.26:6811","nonce":14582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":14582},{"type":"v1","addr":"172.21.15.26:6815","nonce":14582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":14582},{"type":"v1","addr":"172.21.15.26:6813","nonce":14582}]},"public_addr":"172.21.15.26:6809/14582","cluster_addr":"172.21.15.26:6811/14582","heartbeat_back_addr":"172.21.15.26:6815/14582","heartbeat_front_addr":"172.21.15.26:6813/14582","state":["exists","up"]},{"osd":2,"uuid":"8048fbe2-5786-4d39-83a4-853777450c19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":14583},{"type":"v1","addr":"172.21.15.26:6817","nonce":14583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":14583},{"type":"v1","addr":"172.21.15.26:6819","nonce":14583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":14583},{"type":"v1","addr":"172.21.15.26:6823","nonce":14583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":14583},{"type":"v1","addr":"172.21.15.26:6821","nonce":14583}]},"public_addr":"172.21.15.26:6817/14583","cluster_addr":"172.21.15.26:6819/14583","heartbeat_back_addr":"172.21.15.26:6823/14583","heartbeat_front_addr":"172.21.15.26:6821/14583","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:44:33.338318+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:44:33.190657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:44:33.469059+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:44:48.555 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-02-11T09:44:48.555 INFO:tasks.ceph:Creating RBD pool 2021-02-11T09:44:48.555 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster c1 osd pool create rbd 8 2021-02-11T09:44:49.641 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'rbd' created 2021-02-11T09:44:49.654 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-02-11T09:44:50.643 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rbd' on pool 'rbd' 2021-02-11T09:44:50.657 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2021-02-11T09:44:50.658 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-02-11T09:44:50.659 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-02-11T09:44:50.955 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:50.955 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":14,"fsid":"4594769f-0932-4e4a-a0a6-4220b143584b","created":"2021-02-11T09:44:27.643251+0000","modified":"2021-02-11T09:44:50.629843+0000","last_up_change":"2021-02-11T09:44:35.223423+0000","last_in_change":"2021-02-11T09:44:35.223423+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:44:35.233562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T09:44:48.853483+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"73968cb7-66da-4bd6-9b13-f436aa3f1654","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":14584},{"type":"v1","addr":"172.21.15.26:6801","nonce":14584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":14584},{"type":"v1","addr":"172.21.15.26:6803","nonce":14584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":14584},{"type":"v1","addr":"172.21.15.26:6807","nonce":14584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":14584},{"type":"v1","addr":"172.21.15.26:6805","nonce":14584}]},"public_addr":"172.21.15.26:6801/14584","cluster_addr":"172.21.15.26:6803/14584","heartbeat_back_addr":"172.21.15.26:6807/14584","heartbeat_front_addr":"172.21.15.26:6805/14584","state":["exists","up"]},{"osd":1,"uuid":"c3548235-9f89-49fd-a481-722c3fbad94c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":14582},{"type":"v1","addr":"172.21.15.26:6809","nonce":14582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":14582},{"type":"v1","addr":"172.21.15.26:6811","nonce":14582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":14582},{"type":"v1","addr":"172.21.15.26:6815","nonce":14582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":14582},{"type":"v1","addr":"172.21.15.26:6813","nonce":14582}]},"public_addr":"172.21.15.26:6809/14582","cluster_addr":"172.21.15.26:6811/14582","heartbeat_back_addr":"172.21.15.26:6815/14582","heartbeat_front_addr":"172.21.15.26:6813/14582","state":["exists","up"]},{"osd":2,"uuid":"8048fbe2-5786-4d39-83a4-853777450c19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":14583},{"type":"v1","addr":"172.21.15.26:6817","nonce":14583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":14583},{"type":"v1","addr":"172.21.15.26:6819","nonce":14583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":14583},{"type":"v1","addr":"172.21.15.26:6823","nonce":14583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":14583},{"type":"v1","addr":"172.21.15.26:6821","nonce":14583}]},"public_addr":"172.21.15.26:6817/14583","cluster_addr":"172.21.15.26:6819/14583","heartbeat_back_addr":"172.21.15.26:6823/14583","heartbeat_front_addr":"172.21.15.26:6821/14583","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:44:33.338318+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:44:33.190657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:44:33.469059+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:44:50.970 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-11T09:44:35.233562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-02-11T09:44:48.853483+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-11T09:44:50.970 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get device_health_metrics pg_num 2021-02-11T09:44:51.279 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2021-02-11T09:44:51.294 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get rbd pg_num 2021-02-11T09:44:51.606 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2021-02-11T09:44:51.621 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2021-02-11T09:44:51.621 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2021-02-11T09:44:51.621 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2021-02-11T09:44:51.955 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:51.970 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":14398},{"type":"v1","addr":"172.21.15.26:6825","nonce":14398}]},"active_addr":"172.21.15.26:6825/14398","active_change":"2021-02-11T09:44:32.693116+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":707840430}]},{"addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":2768472524}]}]}} 2021-02-11T09:44:51.972 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2021-02-11T09:44:51.972 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2021-02-11T09:44:51.972 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-02-11T09:44:52.275 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:52.275 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":15,"fsid":"4594769f-0932-4e4a-a0a6-4220b143584b","created":"2021-02-11T09:44:27.643251+0000","modified":"2021-02-11T09:44:51.640200+0000","last_up_change":"2021-02-11T09:44:35.223423+0000","last_in_change":"2021-02-11T09:44:35.223423+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:44:35.233562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T09:44:48.853483+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"73968cb7-66da-4bd6-9b13-f436aa3f1654","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":14584},{"type":"v1","addr":"172.21.15.26:6801","nonce":14584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":14584},{"type":"v1","addr":"172.21.15.26:6803","nonce":14584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":14584},{"type":"v1","addr":"172.21.15.26:6807","nonce":14584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":14584},{"type":"v1","addr":"172.21.15.26:6805","nonce":14584}]},"public_addr":"172.21.15.26:6801/14584","cluster_addr":"172.21.15.26:6803/14584","heartbeat_back_addr":"172.21.15.26:6807/14584","heartbeat_front_addr":"172.21.15.26:6805/14584","state":["exists","up"]},{"osd":1,"uuid":"c3548235-9f89-49fd-a481-722c3fbad94c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":14582},{"type":"v1","addr":"172.21.15.26:6809","nonce":14582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":14582},{"type":"v1","addr":"172.21.15.26:6811","nonce":14582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":14582},{"type":"v1","addr":"172.21.15.26:6815","nonce":14582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":14582},{"type":"v1","addr":"172.21.15.26:6813","nonce":14582}]},"public_addr":"172.21.15.26:6809/14582","cluster_addr":"172.21.15.26:6811/14582","heartbeat_back_addr":"172.21.15.26:6815/14582","heartbeat_front_addr":"172.21.15.26:6813/14582","state":["exists","up"]},{"osd":2,"uuid":"8048fbe2-5786-4d39-83a4-853777450c19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":14583},{"type":"v1","addr":"172.21.15.26:6817","nonce":14583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":14583},{"type":"v1","addr":"172.21.15.26:6819","nonce":14583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":14583},{"type":"v1","addr":"172.21.15.26:6823","nonce":14583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":14583},{"type":"v1","addr":"172.21.15.26:6821","nonce":14583}]},"public_addr":"172.21.15.26:6817/14583","cluster_addr":"172.21.15.26:6819/14583","heartbeat_back_addr":"172.21.15.26:6823/14583","heartbeat_front_addr":"172.21.15.26:6821/14583","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:44:33.338318+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:44:33.190657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:44:33.469059+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:44:52.290 INFO:tasks.ceph.ceph_manager.c1:all up! 2021-02-11T09:44:52.290 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-02-11T09:44:52.587 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:52.587 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":15,"fsid":"4594769f-0932-4e4a-a0a6-4220b143584b","created":"2021-02-11T09:44:27.643251+0000","modified":"2021-02-11T09:44:51.640200+0000","last_up_change":"2021-02-11T09:44:35.223423+0000","last_in_change":"2021-02-11T09:44:35.223423+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:44:35.233562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T09:44:48.853483+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"73968cb7-66da-4bd6-9b13-f436aa3f1654","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":14584},{"type":"v1","addr":"172.21.15.26:6801","nonce":14584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":14584},{"type":"v1","addr":"172.21.15.26:6803","nonce":14584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":14584},{"type":"v1","addr":"172.21.15.26:6807","nonce":14584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":14584},{"type":"v1","addr":"172.21.15.26:6805","nonce":14584}]},"public_addr":"172.21.15.26:6801/14584","cluster_addr":"172.21.15.26:6803/14584","heartbeat_back_addr":"172.21.15.26:6807/14584","heartbeat_front_addr":"172.21.15.26:6805/14584","state":["exists","up"]},{"osd":1,"uuid":"c3548235-9f89-49fd-a481-722c3fbad94c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":14582},{"type":"v1","addr":"172.21.15.26:6809","nonce":14582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":14582},{"type":"v1","addr":"172.21.15.26:6811","nonce":14582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":14582},{"type":"v1","addr":"172.21.15.26:6815","nonce":14582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":14582},{"type":"v1","addr":"172.21.15.26:6813","nonce":14582}]},"public_addr":"172.21.15.26:6809/14582","cluster_addr":"172.21.15.26:6811/14582","heartbeat_back_addr":"172.21.15.26:6815/14582","heartbeat_front_addr":"172.21.15.26:6813/14582","state":["exists","up"]},{"osd":2,"uuid":"8048fbe2-5786-4d39-83a4-853777450c19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":14583},{"type":"v1","addr":"172.21.15.26:6817","nonce":14583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":14583},{"type":"v1","addr":"172.21.15.26:6819","nonce":14583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":14583},{"type":"v1","addr":"172.21.15.26:6823","nonce":14583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":14583},{"type":"v1","addr":"172.21.15.26:6821","nonce":14583}]},"public_addr":"172.21.15.26:6817/14583","cluster_addr":"172.21.15.26:6819/14583","heartbeat_back_addr":"172.21.15.26:6823/14583","heartbeat_front_addr":"172.21.15.26:6821/14583","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:44:33.338318+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:44:33.190657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:44:33.469059+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:44:52.602 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2021-02-11T09:44:52.777 INFO:teuthology.orchestra.run.smithi026.stdout:38654705670 2021-02-11T09:44:52.777 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2021-02-11T09:44:52.947 INFO:teuthology.orchestra.run.smithi026.stdout:34359738374 2021-02-11T09:44:52.947 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2021-02-11T09:44:53.121 INFO:teuthology.orchestra.run.smithi026.stdout:38654705671 2021-02-11T09:44:53.121 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-02-11T09:44:53.431 INFO:teuthology.orchestra.run.smithi026.stdout:38654705668 2021-02-11T09:44:53.445 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705670 got 38654705668 for osd.0 2021-02-11T09:44:54.446 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-02-11T09:44:54.760 INFO:teuthology.orchestra.run.smithi026.stdout:38654705669 2021-02-11T09:44:54.779 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705670 got 38654705669 for osd.0 2021-02-11T09:44:55.780 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-02-11T09:44:56.094 INFO:teuthology.orchestra.run.smithi026.stdout:38654705670 2021-02-11T09:44:56.109 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705670 got 38654705670 for osd.0 2021-02-11T09:44:56.109 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2021-02-11T09:44:56.410 INFO:teuthology.orchestra.run.smithi026.stdout:34359738374 2021-02-11T09:44:56.425 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738374 got 34359738374 for osd.1 2021-02-11T09:44:56.426 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2021-02-11T09:44:56.731 INFO:teuthology.orchestra.run.smithi026.stdout:38654705671 2021-02-11T09:44:56.746 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705671 got 38654705671 for osd.2 2021-02-11T09:44:56.746 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2021-02-11T09:44:56.746 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-02-11T09:44:57.035 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:57.038 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2021-02-11T09:44:57.055 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2021-02-11T09:44:56.707115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14808,"kb_used_data":384,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314557992,"statfs":{"total":322122547200,"available":322107383808,"internally_reserved":0,"allocated":393216,"data_stored":112545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:44:50.650752+0000","last_change":"2021-02-11T09:44:50.646966+0000","last_active":"2021-02-11T09:44:50.650752+0000","last_peered":"2021-02-11T09:44:50.650752+0000","last_clean":"2021-02-11T09:44:50.650752+0000","last_became_active":"2021-02-11T09:44:50.646455+0000","last_became_peered":"2021-02-11T09:44:50.646455+0000","last_unstale":"2021-02-11T09:44:50.650752+0000","last_undegraded":"2021-02-11T09:44:50.650752+0000","last_fullsized":"2021-02-11T09:44:50.650752+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_clean_scrub_stamp":"2021-02-11T09:44:49.624422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:44:50.653342+0000","last_change":"2021-02-11T09:44:50.650123+0000","last_active":"2021-02-11T09:44:50.653342+0000","last_peered":"2021-02-11T09:44:50.653342+0000","last_clean":"2021-02-11T09:44:50.653342+0000","last_became_active":"2021-02-11T09:44:50.649772+0000","last_became_peered":"2021-02-11T09:44:50.649772+0000","last_unstale":"2021-02-11T09:44:50.653342+0000","last_undegraded":"2021-02-11T09:44:50.653342+0000","last_fullsized":"2021-02-11T09:44:50.653342+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_clean_scrub_stamp":"2021-02-11T09:44:49.624422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:44:50.653350+0000","last_change":"2021-02-11T09:44:50.649698+0000","last_active":"2021-02-11T09:44:50.653350+0000","last_peered":"2021-02-11T09:44:50.653350+0000","last_clean":"2021-02-11T09:44:50.653350+0000","last_became_active":"2021-02-11T09:44:50.649122+0000","last_became_peered":"2021-02-11T09:44:50.649122+0000","last_unstale":"2021-02-11T09:44:50.653350+0000","last_undegraded":"2021-02-11T09:44:50.653350+0000","last_fullsized":"2021-02-11T09:44:50.653350+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_clean_scrub_stamp":"2021-02-11T09:44:49.624422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:44:50.653274+0000","last_change":"2021-02-11T09:44:50.649207+0000","last_active":"2021-02-11T09:44:50.653274+0000","last_peered":"2021-02-11T09:44:50.653274+0000","last_clean":"2021-02-11T09:44:50.653274+0000","last_became_active":"2021-02-11T09:44:50.648858+0000","last_became_peered":"2021-02-11T09:44:50.648858+0000","last_unstale":"2021-02-11T09:44:50.653274+0000","last_undegraded":"2021-02-11T09:44:50.653274+0000","last_fullsized":"2021-02-11T09:44:50.653274+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_clean_scrub_stamp":"2021-02-11T09:44:49.624422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"12","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:44:50.634842+0000","last_change":"2021-02-11T09:44:37.455841+0000","last_active":"2021-02-11T09:44:50.634842+0000","last_peered":"2021-02-11T09:44:50.634842+0000","last_clean":"2021-02-11T09:44:50.634842+0000","last_became_active":"2021-02-11T09:44:37.454849+0000","last_became_peered":"2021-02-11T09:44:37.454849+0000","last_unstale":"2021-02-11T09:44:50.634842+0000","last_undegraded":"2021-02-11T09:44:50.634842+0000","last_fullsized":"2021-02-11T09:44:50.634842+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:44:36.224975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:44:36.224975+0000","last_clean_scrub_stamp":"2021-02-11T09:44:36.224975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:44:50.650804+0000","last_change":"2021-02-11T09:44:50.647350+0000","last_active":"2021-02-11T09:44:50.650804+0000","last_peered":"2021-02-11T09:44:50.650804+0000","last_clean":"2021-02-11T09:44:50.650804+0000","last_became_active":"2021-02-11T09:44:50.646549+0000","last_became_peered":"2021-02-11T09:44:50.646549+0000","last_unstale":"2021-02-11T09:44:50.650804+0000","last_undegraded":"2021-02-11T09:44:50.650804+0000","last_fullsized":"2021-02-11T09:44:50.650804+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_clean_scrub_stamp":"2021-02-11T09:44:49.624422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:44:50.648393+0000","last_change":"2021-02-11T09:44:50.646066+0000","last_active":"2021-02-11T09:44:50.648393+0000","last_peered":"2021-02-11T09:44:50.648393+0000","last_clean":"2021-02-11T09:44:50.648393+0000","last_became_active":"2021-02-11T09:44:50.645765+0000","last_became_peered":"2021-02-11T09:44:50.645765+0000","last_unstale":"2021-02-11T09:44:50.648393+0000","last_undegraded":"2021-02-11T09:44:50.648393+0000","last_fullsized":"2021-02-11T09:44:50.648393+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_clean_scrub_stamp":"2021-02-11T09:44:49.624422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:44:50.653269+0000","last_change":"2021-02-11T09:44:50.649354+0000","last_active":"2021-02-11T09:44:50.653269+0000","last_peered":"2021-02-11T09:44:50.653269+0000","last_clean":"2021-02-11T09:44:50.653269+0000","last_became_active":"2021-02-11T09:44:50.649009+0000","last_became_peered":"2021-02-11T09:44:50.649009+0000","last_unstale":"2021-02-11T09:44:50.653269+0000","last_undegraded":"2021-02-11T09:44:50.653269+0000","last_fullsized":"2021-02-11T09:44:50.653269+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_clean_scrub_stamp":"2021-02-11T09:44:49.624422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:44:50.653382+0000","last_change":"2021-02-11T09:44:50.649451+0000","last_active":"2021-02-11T09:44:50.653382+0000","last_peered":"2021-02-11T09:44:50.653382+0000","last_clean":"2021-02-11T09:44:50.653382+0000","last_became_active":"2021-02-11T09:44:50.648928+0000","last_became_peered":"2021-02-11T09:44:50.648928+0000","last_unstale":"2021-02-11T09:44:50.653382+0000","last_undegraded":"2021-02-11T09:44:50.653382+0000","last_fullsized":"2021-02-11T09:44:50.653382+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_clean_scrub_stamp":"2021-02-11T09:44:49.624422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705672,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":37515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1779999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":37515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.121}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":37515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.633}]}]}]}} 2021-02-11T09:44:57.055 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-02-11T09:44:57.351 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:57.353 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2021-02-11T09:44:57.370 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2021-02-11T09:44:56.707115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14808,"kb_used_data":384,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314557992,"statfs":{"total":322122547200,"available":322107383808,"internally_reserved":0,"allocated":393216,"data_stored":112545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:44:50.650752+0000","last_change":"2021-02-11T09:44:50.646966+0000","last_active":"2021-02-11T09:44:50.650752+0000","last_peered":"2021-02-11T09:44:50.650752+0000","last_clean":"2021-02-11T09:44:50.650752+0000","last_became_active":"2021-02-11T09:44:50.646455+0000","last_became_peered":"2021-02-11T09:44:50.646455+0000","last_unstale":"2021-02-11T09:44:50.650752+0000","last_undegraded":"2021-02-11T09:44:50.650752+0000","last_fullsized":"2021-02-11T09:44:50.650752+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_clean_scrub_stamp":"2021-02-11T09:44:49.624422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:44:50.653342+0000","last_change":"2021-02-11T09:44:50.650123+0000","last_active":"2021-02-11T09:44:50.653342+0000","last_peered":"2021-02-11T09:44:50.653342+0000","last_clean":"2021-02-11T09:44:50.653342+0000","last_became_active":"2021-02-11T09:44:50.649772+0000","last_became_peered":"2021-02-11T09:44:50.649772+0000","last_unstale":"2021-02-11T09:44:50.653342+0000","last_undegraded":"2021-02-11T09:44:50.653342+0000","last_fullsized":"2021-02-11T09:44:50.653342+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_clean_scrub_stamp":"2021-02-11T09:44:49.624422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:44:50.653350+0000","last_change":"2021-02-11T09:44:50.649698+0000","last_active":"2021-02-11T09:44:50.653350+0000","last_peered":"2021-02-11T09:44:50.653350+0000","last_clean":"2021-02-11T09:44:50.653350+0000","last_became_active":"2021-02-11T09:44:50.649122+0000","last_became_peered":"2021-02-11T09:44:50.649122+0000","last_unstale":"2021-02-11T09:44:50.653350+0000","last_undegraded":"2021-02-11T09:44:50.653350+0000","last_fullsized":"2021-02-11T09:44:50.653350+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_clean_scrub_stamp":"2021-02-11T09:44:49.624422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:44:50.653274+0000","last_change":"2021-02-11T09:44:50.649207+0000","last_active":"2021-02-11T09:44:50.653274+0000","last_peered":"2021-02-11T09:44:50.653274+0000","last_clean":"2021-02-11T09:44:50.653274+0000","last_became_active":"2021-02-11T09:44:50.648858+0000","last_became_peered":"2021-02-11T09:44:50.648858+0000","last_unstale":"2021-02-11T09:44:50.653274+0000","last_undegraded":"2021-02-11T09:44:50.653274+0000","last_fullsized":"2021-02-11T09:44:50.653274+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_clean_scrub_stamp":"2021-02-11T09:44:49.624422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"12","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:44:50.634842+0000","last_change":"2021-02-11T09:44:37.455841+0000","last_active":"2021-02-11T09:44:50.634842+0000","last_peered":"2021-02-11T09:44:50.634842+0000","last_clean":"2021-02-11T09:44:50.634842+0000","last_became_active":"2021-02-11T09:44:37.454849+0000","last_became_peered":"2021-02-11T09:44:37.454849+0000","last_unstale":"2021-02-11T09:44:50.634842+0000","last_undegraded":"2021-02-11T09:44:50.634842+0000","last_fullsized":"2021-02-11T09:44:50.634842+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:44:36.224975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:44:36.224975+0000","last_clean_scrub_stamp":"2021-02-11T09:44:36.224975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:44:50.650804+0000","last_change":"2021-02-11T09:44:50.647350+0000","last_active":"2021-02-11T09:44:50.650804+0000","last_peered":"2021-02-11T09:44:50.650804+0000","last_clean":"2021-02-11T09:44:50.650804+0000","last_became_active":"2021-02-11T09:44:50.646549+0000","last_became_peered":"2021-02-11T09:44:50.646549+0000","last_unstale":"2021-02-11T09:44:50.650804+0000","last_undegraded":"2021-02-11T09:44:50.650804+0000","last_fullsized":"2021-02-11T09:44:50.650804+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_clean_scrub_stamp":"2021-02-11T09:44:49.624422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:44:50.648393+0000","last_change":"2021-02-11T09:44:50.646066+0000","last_active":"2021-02-11T09:44:50.648393+0000","last_peered":"2021-02-11T09:44:50.648393+0000","last_clean":"2021-02-11T09:44:50.648393+0000","last_became_active":"2021-02-11T09:44:50.645765+0000","last_became_peered":"2021-02-11T09:44:50.645765+0000","last_unstale":"2021-02-11T09:44:50.648393+0000","last_undegraded":"2021-02-11T09:44:50.648393+0000","last_fullsized":"2021-02-11T09:44:50.648393+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_clean_scrub_stamp":"2021-02-11T09:44:49.624422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:44:50.653269+0000","last_change":"2021-02-11T09:44:50.649354+0000","last_active":"2021-02-11T09:44:50.653269+0000","last_peered":"2021-02-11T09:44:50.653269+0000","last_clean":"2021-02-11T09:44:50.653269+0000","last_became_active":"2021-02-11T09:44:50.649009+0000","last_became_peered":"2021-02-11T09:44:50.649009+0000","last_unstale":"2021-02-11T09:44:50.653269+0000","last_undegraded":"2021-02-11T09:44:50.653269+0000","last_fullsized":"2021-02-11T09:44:50.653269+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_clean_scrub_stamp":"2021-02-11T09:44:49.624422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:44:50.653382+0000","last_change":"2021-02-11T09:44:50.649451+0000","last_active":"2021-02-11T09:44:50.653382+0000","last_peered":"2021-02-11T09:44:50.653382+0000","last_clean":"2021-02-11T09:44:50.653382+0000","last_became_active":"2021-02-11T09:44:50.648928+0000","last_became_peered":"2021-02-11T09:44:50.648928+0000","last_unstale":"2021-02-11T09:44:50.653382+0000","last_undegraded":"2021-02-11T09:44:50.653382+0000","last_fullsized":"2021-02-11T09:44:50.653382+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:44:49.624422+0000","last_clean_scrub_stamp":"2021-02-11T09:44:49.624422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705672,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":37515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1779999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":37515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.121}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":37515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.633}]}]}]}} 2021-02-11T09:44:57.370 INFO:tasks.ceph.ceph_manager.c1:clean! 2021-02-11T09:44:57.371 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2021-02-11T09:44:57.371 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2021-02-11T09:44:57.371 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2021-02-11T09:44:57.718 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:57.732 INFO:teuthology.orchestra.run.smithi026.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-02-11T09:44:57.732 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2021-02-11T09:44:57.732 INFO:teuthology.run_tasks:Running task ceph... 2021-02-11T09:44:57.740 INFO:tasks.ceph:Creating ceph cluster c2... 2021-02-11T09:44:57.740 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-02-11T09:44:57.740 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874503', 'branch': 'wip-blurpletastic', 'description': 'rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'aemerson@redhat.com', 'first_in_suite': False, 'job_id': '5874503', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-blurpletastic'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f'}}, 'rgw': {'compression type': 'random', 'frontend': 'civetweb'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-blurpletastic', 'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f'}}, 'owner': 'scheduled_aemerson@teuthology', 'priority': 70, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-blurpletastic', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f', 'targets': {'smithi026.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBDwsvOghUsw0AfpZm6OwXwfU7McUxVq6l7RtzNuUCIs2rKPswTlou++DeBtTVmRqJTIvOyW9OoQsjokpItJ041v5vF6BHqilMGhMlgYOzTf38D5IDODjNCE2gWbGEQWgT9RLoBsSxSboqj4JGkGb5RlJorJkG79sce4RT+vjRaDxX3SPXlEGhut2cND0DigGU2o95eMaY+el+7ro58L3cGymLBEQBgdqAG0ylRGOWaJs2IWRQefwZOOt9UMbQNWtvXOd8CQb6B/JVaqj5OAJoRkrbQY2E5V9/oS7kc8EH8EsNWHyStaHB9lSblIVsDAapREJ+ceLCtJZ4vSlLimBp', 'smithi133.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDApmU2RgrDF6OBBSRN7KgTBoRZ18J5CDi+nwxXLLj06kaIh8EVEmOUck98z8UUzSnuJOxY3IzU7xPZwgg8CXnqTn3rdQl6qIh3PU0VReuWmSqZ07BCWiclTLDXQ0ldeDP/WUTKGHVR8FBsCEjOa1Ns7DP0txtTPVkjpRQhTW96V0oWdP9V5NcKrFpOKnkxrQANTim+ih1cMw5r9mBTQQT7IhHS1c5hwAe7JhKIVP+Ur6HhsCgltW9hfC4Wt686bNcI1HHEDA2SI7xQShPKbLq32LP59ifbRaSe5j8voANtgFnK/R2cSW/sonM55p7BRgZlNow4qATJS09nrT3XbdyF'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2021-02-11_09:24:13', 'tube': 'smithi', 'user': 'aemerson', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.29888'} 2021-02-11T09:44:57.741 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-02-11T09:44:57.743 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-02-11T09:44:57.750 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/run/ceph 2021-02-11T09:44:57.789 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -d -m0777 -- /var/run/ceph 2021-02-11T09:44:57.806 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:44:57.806 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/scratch_devs of=/dev/stdout 2021-02-11T09:44:57.853 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-11T09:44:57.853 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-11T09:44:57.902 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-02-11T09:44:57.902 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T09:44:57.902 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 6h/6d Inode: 343 Links: 1 2021-02-11T09:44:57.902 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T09:44:57.902 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2021-02-11 09:41:04.904751917 +0000 2021-02-11T09:44:57.903 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2021-02-11 09:41:04.792807902 +0000 2021-02-11T09:44:57.903 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2021-02-11 09:41:04.792807902 +0000 2021-02-11T09:44:57.903 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2021-02-11T09:44:57.911 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2021-02-11T09:44:57.912 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2021-02-11T09:44:57.912 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000344152 s, 1.5 MB/s 2021-02-11T09:44:57.916 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-11T09:44:57.966 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-02-11T09:44:57.966 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T09:44:57.966 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 6h/6d Inode: 350 Links: 1 2021-02-11T09:44:57.966 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T09:44:57.966 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2021-02-11 09:41:05.116645946 +0000 2021-02-11T09:44:57.966 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2021-02-11 09:41:05.044681936 +0000 2021-02-11T09:44:57.966 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2021-02-11 09:41:05.044681936 +0000 2021-02-11T09:44:57.966 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2021-02-11T09:44:57.975 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2021-02-11T09:44:57.975 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2021-02-11T09:44:57.975 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000393763 s, 1.3 MB/s 2021-02-11T09:44:57.978 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-11T09:44:58.026 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-02-11T09:44:58.026 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T09:44:58.026 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 6h/6d Inode: 356 Links: 1 2021-02-11T09:44:58.026 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T09:44:58.026 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2021-02-11 09:41:05.332537975 +0000 2021-02-11T09:44:58.026 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2021-02-11 09:41:05.232587961 +0000 2021-02-11T09:44:58.026 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2021-02-11 09:41:05.232587961 +0000 2021-02-11T09:44:58.026 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2021-02-11T09:44:58.035 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2021-02-11T09:44:58.035 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2021-02-11T09:44:58.036 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000348702 s, 1.5 MB/s 2021-02-11T09:44:58.040 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-11T09:44:58.090 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-02-11T09:44:58.090 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T09:44:58.090 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 6h/6d Inode: 362 Links: 1 2021-02-11T09:44:58.090 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T09:44:58.090 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2021-02-11 09:41:05.452477991 +0000 2021-02-11T09:44:58.090 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2021-02-11 09:41:05.448479990 +0000 2021-02-11T09:44:58.090 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2021-02-11 09:41:05.448479990 +0000 2021-02-11T09:44:58.091 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2021-02-11T09:44:58.099 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2021-02-11T09:44:58.099 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2021-02-11T09:44:58.099 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000328264 s, 1.6 MB/s 2021-02-11T09:44:58.102 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T09:44:58.102 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi133.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-11T09:44:58.102 INFO:tasks.ceph:Generating config... 2021-02-11T09:44:58.104 INFO:tasks.ceph:[client] debug rgw = 20 2021-02-11T09:44:58.104 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-02-11T09:44:58.104 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-02-11T09:44:58.104 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T09:44:58.104 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-02-11T09:44:58.104 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-02-11T09:44:58.104 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-02-11T09:44:58.104 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-02-11T09:44:58.104 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-02-11T09:44:58.104 INFO:tasks.ceph:[client] setgroup = ceph 2021-02-11T09:44:58.104 INFO:tasks.ceph:[client] setuser = ceph 2021-02-11T09:44:58.104 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2021-02-11T09:44:58.104 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-02-11T09:44:58.105 INFO:tasks.ceph:[mgr] debug ms = 1 2021-02-11T09:44:58.105 INFO:tasks.ceph:[mon] debug mon = 20 2021-02-11T09:44:58.105 INFO:tasks.ceph:[mon] debug ms = 1 2021-02-11T09:44:58.105 INFO:tasks.ceph:[mon] debug paxos = 20 2021-02-11T09:44:58.105 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2021-02-11T09:44:58.105 INFO:tasks.ceph:[osd] debug filestore = 20 2021-02-11T09:44:58.105 INFO:tasks.ceph:[osd] debug journal = 20 2021-02-11T09:44:58.105 INFO:tasks.ceph:[osd] debug ms = 20 2021-02-11T09:44:58.105 INFO:tasks.ceph:[osd] debug osd = 25 2021-02-11T09:44:58.105 INFO:tasks.ceph:[osd] osd fast shutdown = False 2021-02-11T09:44:58.105 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-02-11T09:44:58.105 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-02-11T09:44:58.105 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-02-11T09:44:58.106 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-02-11T09:44:58.106 INFO:tasks.ceph:Setting up c2.mon.a... 2021-02-11T09:44:58.106 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2021-02-11T09:44:58.175 INFO:teuthology.orchestra.run.smithi133.stdout:creating /etc/ceph/c2.keyring 2021-02-11T09:44:58.177 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2021-02-11T09:44:58.254 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-02-11T09:44:58.309 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.133')] 2021-02-11T09:44:58.310 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.133', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-02-11T09:44:58.311 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:44:58.311 DEBUG:teuthology.orchestra.run.smithi133:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-02-11T09:44:58.357 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.133 --print /home/ubuntu/cephtest/c2.monmap 2021-02-11T09:44:58.429 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:44:58.429 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:44:58.430 INFO:teuthology.orchestra.run.smithi133.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2021-02-11T09:44:58.430 INFO:teuthology.orchestra.run.smithi133.stdout:monmaptool: generated fsid 4cba45bb-28f9-46d3-a27e-3c23d56a54ef 2021-02-11T09:44:58.430 INFO:teuthology.orchestra.run.smithi133.stdout:epoch 0 2021-02-11T09:44:58.430 INFO:teuthology.orchestra.run.smithi133.stdout:fsid 4cba45bb-28f9-46d3-a27e-3c23d56a54ef 2021-02-11T09:44:58.430 INFO:teuthology.orchestra.run.smithi133.stdout:last_changed 2021-02-11T09:44:58.433536+0000 2021-02-11T09:44:58.430 INFO:teuthology.orchestra.run.smithi133.stdout:created 2021-02-11T09:44:58.433536+0000 2021-02-11T09:44:58.430 INFO:teuthology.orchestra.run.smithi133.stdout:min_mon_release 0 (unknown) 2021-02-11T09:44:58.430 INFO:teuthology.orchestra.run.smithi133.stdout:election_strategy: 1 2021-02-11T09:44:58.430 INFO:teuthology.orchestra.run.smithi133.stdout:0: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.a 2021-02-11T09:44:58.430 INFO:teuthology.orchestra.run.smithi133.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2021-02-11T09:44:58.431 DEBUG:teuthology.orchestra.run.smithi133:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-02-11T09:44:58.481 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 4cba45bb-28f9-46d3-a27e-3c23d56a54ef... 2021-02-11T09:44:58.481 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-02-11T09:44:58.483 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-02-11T09:44:58.526 INFO:teuthology.orchestra.run.smithi026.stdout:[global] 2021-02-11T09:44:58.526 INFO:teuthology.orchestra.run.smithi026.stdout: chdir = "" 2021-02-11T09:44:58.526 INFO:teuthology.orchestra.run.smithi026.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-11T09:44:58.526 INFO:teuthology.orchestra.run.smithi026.stdout: auth supported = cephx 2021-02-11T09:44:58.526 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: filestore xattr use omap = true 2021-02-11T09:44:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: mon clock drift allowed = 1.000 2021-02-11T09:44:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: osd crush chooseleaf type = 0 2021-02-11T09:44:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: auth debug = true 2021-02-11T09:44:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on old message = true 2021-02-11T09:44:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on bug = true 2021-02-11T09:44:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-11T09:44:58.527 INFO:teuthology.orchestra.run.smithi026.stdout: mon pg warn max object skew = 0 2021-02-11T09:44:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: # disable pg_autoscaler by default for new pools 2021-02-11T09:44:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-11T09:44:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default size = 2 2021-02-11T09:44:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow primary affinity = true 2021-02-11T09:44:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow pg remap = true 2021-02-11T09:44:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on legacy crush tunables = false 2021-02-11T09:44:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on crush straw calc version zero = false 2021-02-11T09:44:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on no sortbitwise = false 2021-02-11T09:44:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on osd down out interval zero = false 2021-02-11T09:44:58.528 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on too few osds = false 2021-02-11T09:44:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-11T09:44:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-11T09:44:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: mon_allow_pool_size_one = true 2021-02-11T09:44:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-11T09:44:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: osd default data pool replay window = 5 2021-02-11T09:44:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: mon allow pool delete = true 2021-02-11T09:44:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: mon cluster log file level = debug 2021-02-11T09:44:58.529 INFO:teuthology.orchestra.run.smithi026.stdout: debug asserts on shutdown = true 2021-02-11T09:44:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: mon health detail to clog = false 2021-02-11T09:44:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: mon host = 172.21.15.133 2021-02-11T09:44:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: osd heartbeat grace = 40 2021-02-11T09:44:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: fsid = 4cba45bb-28f9-46d3-a27e-3c23d56a54ef 2021-02-11T09:44:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.530 INFO:teuthology.orchestra.run.smithi026.stdout:[osd] 2021-02-11T09:44:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: osd journal size = 100 2021-02-11T09:44:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub load threshold = 5.0 2021-02-11T09:44:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub max interval = 600 2021-02-11T09:44:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.530 INFO:teuthology.orchestra.run.smithi026.stdout: osd recover clone overlap = true 2021-02-11T09:44:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: osd recovery max chunk = 1048576 2021-02-11T09:44:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug shutdown = true 2021-02-11T09:44:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug op order = true 2021-02-11T09:44:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify stray on activate = true 2021-02-11T09:44:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: osd open classes on start = true 2021-02-11T09:44:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug pg log writeout = true 2021-02-11T09:44:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: osd deep scrub update digest min age = 30 2021-02-11T09:44:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.531 INFO:teuthology.orchestra.run.smithi026.stdout: osd map max advance = 10 2021-02-11T09:44:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: journal zero on create = true 2021-02-11T09:44:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: filestore ondisk finisher threads = 3 2021-02-11T09:44:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: filestore apply finisher threads = 3 2021-02-11T09:44:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: bdev debug aio = true 2021-02-11T09:44:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug misdirected ops = true 2021-02-11T09:44:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: debug filestore = 20 2021-02-11T09:44:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: debug journal = 20 2021-02-11T09:44:58.532 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 20 2021-02-11T09:44:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: debug osd = 25 2021-02-11T09:44:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: osd fast shutdown = False 2021-02-11T09:44:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: rocksdb delete range threshold = 0 2021-02-11T09:44:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.533 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr] 2021-02-11T09:44:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2021-02-11T09:44:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: debug mgr = 20 2021-02-11T09:44:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2021-02-11T09:44:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2021-02-11T09:44:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2021-02-11T09:44:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2021-02-11T09:44:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: mgr/telemetry/nag = false 2021-02-11T09:44:58.533 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.534 INFO:teuthology.orchestra.run.smithi026.stdout:[mon] 2021-02-11T09:44:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2021-02-11T09:44:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2021-02-11T09:44:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: debug paxos = 20 2021-02-11T09:44:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2021-02-11T09:44:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: mon data avail warn = 5 2021-02-11T09:44:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: mon mgr mkfs grace = 240 2021-02-11T09:44:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2021-02-11T09:44:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd reporter subtree level = osd 2021-02-11T09:44:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd prime pg temp = true 2021-02-11T09:44:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2021-02-11T09:44:58.534 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd crush smoke test = False 2021-02-11T09:44:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T09:44:58.535 INFO:teuthology.orchestra.run.smithi026.stdout:[client] 2021-02-11T09:44:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: rgw cache enabled = true 2021-02-11T09:44:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable ops log = true 2021-02-11T09:44:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable usage log = true 2021-02-11T09:44:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-11T09:44:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-11T09:44:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: debug rgw = 20 2021-02-11T09:44:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: rgw crypt require ssl = False 2021-02-11T09:44:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: rgw crypt s3 kms backend = testing 2021-02-11T09:44:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T09:44:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: rgw curl low speed time = 300 2021-02-11T09:44:58.535 INFO:teuthology.orchestra.run.smithi026.stdout: rgw data log num shards = 4 2021-02-11T09:44:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: rgw md log max shards = 4 2021-02-11T09:44:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: rgw sync log trim interval = 0 2021-02-11T09:44:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: rgw sync obj etag verify = True 2021-02-11T09:44:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: setgroup = ceph 2021-02-11T09:44:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: setuser = ceph 2021-02-11T09:44:58.536 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.a] 2021-02-11T09:44:58.536 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.0] 2021-02-11T09:44:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: osd_max_omap_entries_per_request = 10 2021-02-11T09:44:58.536 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.1] 2021-02-11T09:44:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: osd_max_omap_entries_per_request = 1000 2021-02-11T09:44:58.536 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.2] 2021-02-11T09:44:58.536 INFO:teuthology.orchestra.run.smithi026.stdout: osd_max_omap_entries_per_request = 10000 2021-02-11T09:44:58.554 INFO:teuthology.orchestra.run.smithi133.stdout:[global] 2021-02-11T09:44:58.554 INFO:teuthology.orchestra.run.smithi133.stdout: chdir = "" 2021-02-11T09:44:58.554 INFO:teuthology.orchestra.run.smithi133.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-11T09:44:58.554 INFO:teuthology.orchestra.run.smithi133.stdout: auth supported = cephx 2021-02-11T09:44:58.554 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.554 INFO:teuthology.orchestra.run.smithi133.stdout: filestore xattr use omap = true 2021-02-11T09:44:58.555 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.555 INFO:teuthology.orchestra.run.smithi133.stdout: mon clock drift allowed = 1.000 2021-02-11T09:44:58.555 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.555 INFO:teuthology.orchestra.run.smithi133.stdout: osd crush chooseleaf type = 0 2021-02-11T09:44:58.555 INFO:teuthology.orchestra.run.smithi133.stdout: auth debug = true 2021-02-11T09:44:58.555 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.555 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on old message = true 2021-02-11T09:44:58.555 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on bug = true 2021-02-11T09:44:58.555 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.555 INFO:teuthology.orchestra.run.smithi133.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-11T09:44:58.555 INFO:teuthology.orchestra.run.smithi133.stdout: mon pg warn max object skew = 0 2021-02-11T09:44:58.555 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.555 INFO:teuthology.orchestra.run.smithi133.stdout: # disable pg_autoscaler by default for new pools 2021-02-11T09:44:58.556 INFO:teuthology.orchestra.run.smithi133.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-11T09:44:58.556 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.556 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default size = 2 2021-02-11T09:44:58.556 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.556 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow primary affinity = true 2021-02-11T09:44:58.556 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow pg remap = true 2021-02-11T09:44:58.556 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on legacy crush tunables = false 2021-02-11T09:44:58.556 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on crush straw calc version zero = false 2021-02-11T09:44:58.556 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on no sortbitwise = false 2021-02-11T09:44:58.556 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on osd down out interval zero = false 2021-02-11T09:44:58.556 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on too few osds = false 2021-02-11T09:44:58.556 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-11T09:44:58.556 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-11T09:44:58.556 INFO:teuthology.orchestra.run.smithi133.stdout: mon_allow_pool_size_one = true 2021-02-11T09:44:58.557 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.557 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-11T09:44:58.557 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.557 INFO:teuthology.orchestra.run.smithi133.stdout: osd default data pool replay window = 5 2021-02-11T09:44:58.557 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.557 INFO:teuthology.orchestra.run.smithi133.stdout: mon allow pool delete = true 2021-02-11T09:44:58.557 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.557 INFO:teuthology.orchestra.run.smithi133.stdout: mon cluster log file level = debug 2021-02-11T09:44:58.557 INFO:teuthology.orchestra.run.smithi133.stdout: debug asserts on shutdown = true 2021-02-11T09:44:58.557 INFO:teuthology.orchestra.run.smithi133.stdout: mon health detail to clog = false 2021-02-11T09:44:58.557 INFO:teuthology.orchestra.run.smithi133.stdout: mon host = 172.21.15.133 2021-02-11T09:44:58.557 INFO:teuthology.orchestra.run.smithi133.stdout: osd heartbeat grace = 40 2021-02-11T09:44:58.557 INFO:teuthology.orchestra.run.smithi133.stdout: fsid = 4cba45bb-28f9-46d3-a27e-3c23d56a54ef 2021-02-11T09:44:58.557 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.558 INFO:teuthology.orchestra.run.smithi133.stdout:[osd] 2021-02-11T09:44:58.558 INFO:teuthology.orchestra.run.smithi133.stdout: osd journal size = 100 2021-02-11T09:44:58.558 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.558 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub load threshold = 5.0 2021-02-11T09:44:58.558 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub max interval = 600 2021-02-11T09:44:58.558 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.558 INFO:teuthology.orchestra.run.smithi133.stdout: osd recover clone overlap = true 2021-02-11T09:44:58.558 INFO:teuthology.orchestra.run.smithi133.stdout: osd recovery max chunk = 1048576 2021-02-11T09:44:58.558 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.558 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug shutdown = true 2021-02-11T09:44:58.558 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug op order = true 2021-02-11T09:44:58.558 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug verify stray on activate = true 2021-02-11T09:44:58.559 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.559 INFO:teuthology.orchestra.run.smithi133.stdout: osd open classes on start = true 2021-02-11T09:44:58.559 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug pg log writeout = true 2021-02-11T09:44:58.559 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.559 INFO:teuthology.orchestra.run.smithi133.stdout: osd deep scrub update digest min age = 30 2021-02-11T09:44:58.559 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.559 INFO:teuthology.orchestra.run.smithi133.stdout: osd map max advance = 10 2021-02-11T09:44:58.559 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.559 INFO:teuthology.orchestra.run.smithi133.stdout: journal zero on create = true 2021-02-11T09:44:58.559 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.559 INFO:teuthology.orchestra.run.smithi133.stdout: filestore ondisk finisher threads = 3 2021-02-11T09:44:58.559 INFO:teuthology.orchestra.run.smithi133.stdout: filestore apply finisher threads = 3 2021-02-11T09:44:58.559 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.559 INFO:teuthology.orchestra.run.smithi133.stdout: bdev debug aio = true 2021-02-11T09:44:58.560 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug misdirected ops = true 2021-02-11T09:44:58.560 INFO:teuthology.orchestra.run.smithi133.stdout: debug filestore = 20 2021-02-11T09:44:58.560 INFO:teuthology.orchestra.run.smithi133.stdout: debug journal = 20 2021-02-11T09:44:58.560 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 20 2021-02-11T09:44:58.560 INFO:teuthology.orchestra.run.smithi133.stdout: debug osd = 25 2021-02-11T09:44:58.560 INFO:teuthology.orchestra.run.smithi133.stdout: osd fast shutdown = False 2021-02-11T09:44:58.560 INFO:teuthology.orchestra.run.smithi133.stdout: rocksdb delete range threshold = 0 2021-02-11T09:44:58.560 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.560 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr] 2021-02-11T09:44:58.560 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2021-02-11T09:44:58.560 INFO:teuthology.orchestra.run.smithi133.stdout: debug mgr = 20 2021-02-11T09:44:58.560 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2021-02-11T09:44:58.560 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2021-02-11T09:44:58.561 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2021-02-11T09:44:58.561 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2021-02-11T09:44:58.561 INFO:teuthology.orchestra.run.smithi133.stdout: mgr/telemetry/nag = false 2021-02-11T09:44:58.561 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.561 INFO:teuthology.orchestra.run.smithi133.stdout:[mon] 2021-02-11T09:44:58.561 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2021-02-11T09:44:58.561 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2021-02-11T09:44:58.561 INFO:teuthology.orchestra.run.smithi133.stdout: debug paxos = 20 2021-02-11T09:44:58.561 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2021-02-11T09:44:58.561 INFO:teuthology.orchestra.run.smithi133.stdout: mon data avail warn = 5 2021-02-11T09:44:58.561 INFO:teuthology.orchestra.run.smithi133.stdout: mon mgr mkfs grace = 240 2021-02-11T09:44:58.561 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2021-02-11T09:44:58.561 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd reporter subtree level = osd 2021-02-11T09:44:58.562 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd prime pg temp = true 2021-02-11T09:44:58.562 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2021-02-11T09:44:58.562 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd crush smoke test = False 2021-02-11T09:44:58.562 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:44:58.562 INFO:teuthology.orchestra.run.smithi133.stdout:[client] 2021-02-11T09:44:58.562 INFO:teuthology.orchestra.run.smithi133.stdout: rgw cache enabled = true 2021-02-11T09:44:58.562 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable ops log = true 2021-02-11T09:44:58.562 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable usage log = true 2021-02-11T09:44:58.562 INFO:teuthology.orchestra.run.smithi133.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-11T09:44:58.562 INFO:teuthology.orchestra.run.smithi133.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-11T09:44:58.562 INFO:teuthology.orchestra.run.smithi133.stdout: debug rgw = 20 2021-02-11T09:44:58.562 INFO:teuthology.orchestra.run.smithi133.stdout: rgw crypt require ssl = False 2021-02-11T09:44:58.562 INFO:teuthology.orchestra.run.smithi133.stdout: rgw crypt s3 kms backend = testing 2021-02-11T09:44:58.563 INFO:teuthology.orchestra.run.smithi133.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T09:44:58.563 INFO:teuthology.orchestra.run.smithi133.stdout: rgw curl low speed time = 300 2021-02-11T09:44:58.563 INFO:teuthology.orchestra.run.smithi133.stdout: rgw data log num shards = 4 2021-02-11T09:44:58.563 INFO:teuthology.orchestra.run.smithi133.stdout: rgw md log max shards = 4 2021-02-11T09:44:58.563 INFO:teuthology.orchestra.run.smithi133.stdout: rgw sync log trim interval = 0 2021-02-11T09:44:58.563 INFO:teuthology.orchestra.run.smithi133.stdout: rgw sync obj etag verify = True 2021-02-11T09:44:58.563 INFO:teuthology.orchestra.run.smithi133.stdout: setgroup = ceph 2021-02-11T09:44:58.563 INFO:teuthology.orchestra.run.smithi133.stdout: setuser = ceph 2021-02-11T09:44:58.563 INFO:teuthology.orchestra.run.smithi133.stdout:[mon.a] 2021-02-11T09:44:58.563 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.0] 2021-02-11T09:44:58.563 INFO:teuthology.orchestra.run.smithi133.stdout: osd_max_omap_entries_per_request = 10 2021-02-11T09:44:58.563 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.1] 2021-02-11T09:44:58.563 INFO:teuthology.orchestra.run.smithi133.stdout: osd_max_omap_entries_per_request = 1000 2021-02-11T09:44:58.563 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.2] 2021-02-11T09:44:58.564 INFO:teuthology.orchestra.run.smithi133.stdout: osd_max_omap_entries_per_request = 10000 2021-02-11T09:44:58.564 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2021-02-11T09:44:58.564 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2021-02-11T09:44:58.652 INFO:tasks.ceph:Copying monmap to all nodes... 2021-02-11T09:44:58.652 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:44:58.652 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2021-02-11T09:44:58.701 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:44:58.701 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2021-02-11T09:44:58.748 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi026.front.sepia.ceph.com 2021-02-11T09:44:58.749 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:58.749 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/c2.keyring 2021-02-11T09:44:58.749 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-02-11T09:44:58.774 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2021-02-11T09:44:58.774 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-02-11T09:44:58.822 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi133.front.sepia.ceph.com 2021-02-11T09:44:58.822 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:44:58.822 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/ceph/c2.keyring 2021-02-11T09:44:58.822 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-02-11T09:44:58.847 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:44:58.847 DEBUG:teuthology.orchestra.run.smithi133:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-02-11T09:44:58.897 INFO:tasks.ceph:Setting up mon nodes... 2021-02-11T09:44:58.897 INFO:tasks.ceph:Setting up mgr nodes... 2021-02-11T09:44:58.897 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2021-02-11T09:44:58.979 INFO:teuthology.orchestra.run.smithi133.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2021-02-11T09:44:58.982 INFO:tasks.ceph:Setting up mds nodes... 2021-02-11T09:44:58.982 INFO:tasks.ceph_client:Setting up client nodes... 2021-02-11T09:44:58.982 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2021-02-11T09:44:59.055 INFO:teuthology.orchestra.run.smithi133.stdout:creating /etc/ceph/c2.client.0.keyring 2021-02-11T09:44:59.068 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2021-02-11T09:44:59.144 INFO:teuthology.orchestra.run.smithi133.stdout:creating /etc/ceph/c2.client.1.keyring 2021-02-11T09:44:59.155 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-02-11T09:44:59.155 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi133.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-11T09:44:59.155 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2021-02-11T09:44:59.213 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T09:44:59.214 INFO:tasks.ceph:role: c2.osd.0 2021-02-11T09:44:59.214 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi133.front.sepia.ceph.com 2021-02-11T09:44:59.214 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-02-11T09:44:59.724 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T09:44:59.724 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T09:44:59.724 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T09:44:59.724 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T09:44:59.724 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2021-02-11T09:44:59.724 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T09:44:59.724 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T09:44:59.724 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T09:44:59.724 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T09:44:59.725 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2021-02-11T09:44:59.726 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2021-02-11T09:44:59.805 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2021-02-11T09:44:59.817 INFO:teuthology.orchestra.run.smithi133.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T09:44:59.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T09:44:59.818 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2021-02-11T09:44:59.872 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T09:44:59.872 INFO:tasks.ceph:role: c2.osd.1 2021-02-11T09:44:59.872 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi133.front.sepia.ceph.com 2021-02-11T09:44:59.873 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-02-11T09:45:00.388 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T09:45:00.388 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T09:45:00.388 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T09:45:00.388 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T09:45:00.388 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2021-02-11T09:45:00.388 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T09:45:00.388 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T09:45:00.388 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T09:45:00.388 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T09:45:00.389 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2021-02-11T09:45:00.390 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2021-02-11T09:45:00.458 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2021-02-11T09:45:00.469 INFO:teuthology.orchestra.run.smithi133.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T09:45:00.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T09:45:00.471 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2021-02-11T09:45:00.524 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T09:45:00.525 INFO:tasks.ceph:role: c2.osd.2 2021-02-11T09:45:00.525 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi133.front.sepia.ceph.com 2021-02-11T09:45:00.525 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-02-11T09:45:01.017 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T09:45:01.017 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T09:45:01.017 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T09:45:01.017 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T09:45:01.017 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2021-02-11T09:45:01.018 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T09:45:01.018 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T09:45:01.018 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T09:45:01.018 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T09:45:01.019 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2021-02-11T09:45:01.019 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2021-02-11T09:45:01.058 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2021-02-11T09:45:01.111 INFO:teuthology.orchestra.run.smithi133.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T09:45:01.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T09:45:01.112 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2021-02-11T09:45:01.295 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:45:01.294+0000 7fe8f6e0a080 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2021-02-11T09:45:01.296 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:45:01.298+0000 7fe8f6e0a080 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2021-02-11T09:45:01.296 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:45:01.298+0000 7fe8f6e0a080 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-02-11T09:45:01.296 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:45:01.298+0000 7fe8f6e0a080 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-02-11T09:45:01.296 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:45:01.298+0000 7fe8f6e0a080 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-02-11T09:45:01.302 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:45:01.302+0000 7fe8f6e0a080 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2021-02-11T09:45:03.480 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2021-02-11T09:45:14.689 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2021-02-11T09:45:14.770 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:45:14.770+0000 7f7e36bb2080 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2021-02-11T09:45:14.770 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:45:14.770+0000 7f7e36bb2080 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2021-02-11T09:45:14.770 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:45:14.770+0000 7f7e36bb2080 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-02-11T09:45:14.770 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:45:14.770+0000 7f7e36bb2080 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-02-11T09:45:14.770 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:45:14.770+0000 7f7e36bb2080 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-02-11T09:45:14.771 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:45:14.770+0000 7f7e36bb2080 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2021-02-11T09:45:16.937 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2021-02-11T09:45:16.988 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2021-02-11T09:45:17.066 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:45:17.065+0000 7f924827f080 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2021-02-11T09:45:17.066 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:45:17.065+0000 7f924827f080 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2021-02-11T09:45:17.066 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:45:17.065+0000 7f924827f080 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-02-11T09:45:17.066 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:45:17.065+0000 7f924827f080 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-02-11T09:45:17.066 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:45:17.065+0000 7f924827f080 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-02-11T09:45:17.067 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:45:17.065+0000 7f924827f080 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2021-02-11T09:45:19.237 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2021-02-11T09:45:19.295 INFO:tasks.ceph:Reading keys from all nodes... 2021-02-11T09:45:19.295 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:45:19.295 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2021-02-11T09:45:19.349 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:45:19.349 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2021-02-11T09:45:19.406 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:45:19.406 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2021-02-11T09:45:19.460 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:45:19.460 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2021-02-11T09:45:19.513 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:45:19.513 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2021-02-11T09:45:19.562 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:45:19.562 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2021-02-11T09:45:19.610 INFO:tasks.ceph:Adding keys to all mons... 2021-02-11T09:45:19.610 DEBUG:teuthology.orchestra.run.smithi133:> sudo tee -a /etc/ceph/c2.keyring 2021-02-11T09:45:19.665 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr.x] 2021-02-11T09:45:19.666 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQCa/CRgIKqcOhAAzdu1IoeuhjUo28Y6qLV0YQ== 2021-02-11T09:45:19.666 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.0] 2021-02-11T09:45:19.666 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQCd/CRgVFvOERAAaIqiCT8ic/DUtnnfiU2Yew== 2021-02-11T09:45:19.666 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.1] 2021-02-11T09:45:19.666 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQCq/CRgXocdLhAAGc667OHxSLw05OnpdjKn2w== 2021-02-11T09:45:19.666 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.2] 2021-02-11T09:45:19.666 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQCt/CRgryclBBAAXsOHbXUw+AQw17NOLOrwKQ== 2021-02-11T09:45:19.666 INFO:teuthology.orchestra.run.smithi133.stdout:[client.0] 2021-02-11T09:45:19.666 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQCb/CRgqQmLAxAAiSJdahGRyA8FNbQWLILl5g== 2021-02-11T09:45:19.666 INFO:teuthology.orchestra.run.smithi133.stdout:[client.1] 2021-02-11T09:45:19.666 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQCb/CRgRuXPCBAAnuk/N8fM6CpXTC1hpC7mJw== 2021-02-11T09:45:19.667 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-11T09:45:19.754 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T09:45:19.795 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T09:45:19.876 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T09:45:19.962 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-11T09:45:20.007 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-11T09:45:20.090 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-02-11T09:45:20.090 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2021-02-11T09:45:20.151 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2021-02-11T09:45:20.264 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2021-02-11T09:45:20.281 DEBUG:teuthology.orchestra.run.smithi133:> rm -- /home/ubuntu/cephtest/c2.monmap 2021-02-11T09:45:20.327 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2021-02-11T09:45:20.327 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2021-02-11T09:45:20.327 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2021-02-11T09:45:20.370 INFO:tasks.ceph.c2.mon.a:Started 2021-02-11T09:45:20.371 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2021-02-11T09:45:20.371 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2021-02-11T09:45:20.371 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2021-02-11T09:45:20.373 INFO:tasks.ceph.c2.mgr.x:Started 2021-02-11T09:45:20.373 INFO:tasks.ceph:Setting crush tunables to default 2021-02-11T09:45:20.373 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c2 osd crush tunables default 2021-02-11T09:45:20.782 INFO:teuthology.orchestra.run.smithi133.stderr:adjusted tunables profile to default 2021-02-11T09:45:20.796 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2021-02-11T09:45:20.796 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:45:20.796 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2021-02-11T09:45:20.805 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:45:20.805 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2021-02-11T09:45:20.861 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2021-02-11T09:45:20.861 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2021-02-11T09:45:20.918 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c2 osd new 1bd20326-5d00-4271-8124-7670b3b725ab 0 2021-02-11T09:45:21.171 INFO:teuthology.orchestra.run.smithi133.stdout:0 2021-02-11T09:45:21.183 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c2 osd new 2e90ab89-8f3b-459a-96ff-00d2d91bac75 1 2021-02-11T09:45:21.406 INFO:teuthology.orchestra.run.smithi133.stdout:1 2021-02-11T09:45:21.419 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c2 osd new 1b98a02b-858a-4100-8da2-2021ed431999 2 2021-02-11T09:45:21.648 INFO:teuthology.orchestra.run.smithi133.stdout:2 2021-02-11T09:45:21.660 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2021-02-11T09:45:21.660 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2021-02-11T09:45:21.662 INFO:tasks.ceph.c2.osd.0:Started 2021-02-11T09:45:21.662 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2021-02-11T09:45:21.662 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2021-02-11T09:45:21.664 INFO:tasks.ceph.c2.osd.1:Started 2021-02-11T09:45:21.665 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2021-02-11T09:45:21.665 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2021-02-11T09:45:21.667 INFO:tasks.ceph.c2.osd.2:Started 2021-02-11T09:45:21.667 INFO:tasks.ceph:Waiting for OSDs to come up 2021-02-11T09:45:21.970 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-02-11T09:45:22.179 INFO:teuthology.misc.health.smithi133.stdout: 2021-02-11T09:45:22.179 INFO:teuthology.misc.health.smithi133.stdout:{"epoch":5,"fsid":"4cba45bb-28f9-46d3-a27e-3c23d56a54ef","created":"2021-02-11T09:45:20.484106+0000","modified":"2021-02-11T09:45:21.647345+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"1bd20326-5d00-4271-8124-7670b3b725ab","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2e90ab89-8f3b-459a-96ff-00d2d91bac75","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b98a02b-858a-4100-8da2-2021ed431999","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:45:22.190 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-02-11T09:45:22.276 INFO:tasks.ceph.c2.osd.2.smithi133.stderr:2021-02-11T09:45:22.275+0000 7f5fea17e080 -1 Falling back to public interface 2021-02-11T09:45:22.311 INFO:tasks.ceph.c2.osd.0.smithi133.stderr:2021-02-11T09:45:22.311+0000 7fbad23df080 -1 Falling back to public interface 2021-02-11T09:45:22.323 INFO:tasks.ceph.c2.osd.1.smithi133.stderr:2021-02-11T09:45:22.323+0000 7f9420756080 -1 Falling back to public interface 2021-02-11T09:45:25.006 INFO:tasks.ceph.c2.osd.2.smithi133.stderr:2021-02-11T09:45:25.002+0000 7f5fea17e080 -1 osd.2 0 log_to_monitors {default=true} 2021-02-11T09:45:25.302 INFO:tasks.ceph.c2.osd.0.smithi133.stderr:2021-02-11T09:45:25.298+0000 7fbad23df080 -1 osd.0 0 log_to_monitors {default=true} 2021-02-11T09:45:25.369 INFO:tasks.ceph.c2.osd.1.smithi133.stderr:2021-02-11T09:45:25.366+0000 7f9420756080 -1 osd.1 0 log_to_monitors {default=true} 2021-02-11T09:45:26.072 INFO:tasks.ceph.c2.osd.2.smithi133.stderr:2021-02-11T09:45:26.070+0000 7f5fe1647700 -1 osd.2 0 waiting for initial osdmap 2021-02-11T09:45:27.079 INFO:tasks.ceph.c2.osd.0.smithi133.stderr:2021-02-11T09:45:27.078+0000 7fbac98a8700 -1 osd.0 0 waiting for initial osdmap 2021-02-11T09:45:27.081 INFO:tasks.ceph.c2.osd.1.smithi133.stderr:2021-02-11T09:45:27.078+0000 7f9419422700 -1 osd.1 0 waiting for initial osdmap 2021-02-11T09:45:28.493 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-02-11T09:45:28.789 INFO:teuthology.misc.health.smithi133.stdout: 2021-02-11T09:45:28.789 INFO:teuthology.misc.health.smithi133.stdout:{"epoch":9,"fsid":"4cba45bb-28f9-46d3-a27e-3c23d56a54ef","created":"2021-02-11T09:45:20.484106+0000","modified":"2021-02-11T09:45:28.077370+0000","last_up_change":"2021-02-11T09:45:28.077370+0000","last_in_change":"2021-02-11T09:45:28.077370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"1bd20326-5d00-4271-8124-7670b3b725ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":14635},{"type":"v1","addr":"172.21.15.133:6809","nonce":14635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":14635},{"type":"v1","addr":"172.21.15.133:6811","nonce":14635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":14635},{"type":"v1","addr":"172.21.15.133:6815","nonce":14635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":14635},{"type":"v1","addr":"172.21.15.133:6813","nonce":14635}]},"public_addr":"172.21.15.133:6809/14635","cluster_addr":"172.21.15.133:6811/14635","heartbeat_back_addr":"172.21.15.133:6815/14635","heartbeat_front_addr":"172.21.15.133:6813/14635","state":["exists","up"]},{"osd":1,"uuid":"2e90ab89-8f3b-459a-96ff-00d2d91bac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":14636},{"type":"v1","addr":"172.21.15.133:6817","nonce":14636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":14636},{"type":"v1","addr":"172.21.15.133:6819","nonce":14636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":14636},{"type":"v1","addr":"172.21.15.133:6823","nonce":14636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":14636},{"type":"v1","addr":"172.21.15.133:6821","nonce":14636}]},"public_addr":"172.21.15.133:6817/14636","cluster_addr":"172.21.15.133:6819/14636","heartbeat_back_addr":"172.21.15.133:6823/14636","heartbeat_front_addr":"172.21.15.133:6821/14636","state":["exists","up"]},{"osd":2,"uuid":"1b98a02b-858a-4100-8da2-2021ed431999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":14637},{"type":"v1","addr":"172.21.15.133:6801","nonce":14637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":14637},{"type":"v1","addr":"172.21.15.133:6803","nonce":14637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":14637},{"type":"v1","addr":"172.21.15.133:6807","nonce":14637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":14637},{"type":"v1","addr":"172.21.15.133:6805","nonce":14637}]},"public_addr":"172.21.15.133:6801/14637","cluster_addr":"172.21.15.133:6803/14637","heartbeat_back_addr":"172.21.15.133:6807/14637","heartbeat_front_addr":"172.21.15.133:6805/14637","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:45:26.018929+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:45:28.802 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-02-11T09:45:28.802 INFO:tasks.ceph:Creating RBD pool 2021-02-11T09:45:28.803 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c2 osd pool create rbd 8 2021-02-11T09:45:30.107 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'rbd' created 2021-02-11T09:45:30.124 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-02-11T09:45:31.135 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rbd' on pool 'rbd' 2021-02-11T09:45:31.144 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2021-02-11T09:45:31.144 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-02-11T09:45:31.145 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-02-11T09:45:31.434 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:45:31.435 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":12,"fsid":"4cba45bb-28f9-46d3-a27e-3c23d56a54ef","created":"2021-02-11T09:45:20.484106+0000","modified":"2021-02-11T09:45:31.105205+0000","last_up_change":"2021-02-11T09:45:28.077370+0000","last_in_change":"2021-02-11T09:45:28.077370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:45:28.093603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T09:45:29.100223+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"1bd20326-5d00-4271-8124-7670b3b725ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":14635},{"type":"v1","addr":"172.21.15.133:6809","nonce":14635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":14635},{"type":"v1","addr":"172.21.15.133:6811","nonce":14635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":14635},{"type":"v1","addr":"172.21.15.133:6815","nonce":14635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":14635},{"type":"v1","addr":"172.21.15.133:6813","nonce":14635}]},"public_addr":"172.21.15.133:6809/14635","cluster_addr":"172.21.15.133:6811/14635","heartbeat_back_addr":"172.21.15.133:6815/14635","heartbeat_front_addr":"172.21.15.133:6813/14635","state":["exists","up"]},{"osd":1,"uuid":"2e90ab89-8f3b-459a-96ff-00d2d91bac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":14636},{"type":"v1","addr":"172.21.15.133:6817","nonce":14636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":14636},{"type":"v1","addr":"172.21.15.133:6819","nonce":14636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":14636},{"type":"v1","addr":"172.21.15.133:6823","nonce":14636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":14636},{"type":"v1","addr":"172.21.15.133:6821","nonce":14636}]},"public_addr":"172.21.15.133:6817/14636","cluster_addr":"172.21.15.133:6819/14636","heartbeat_back_addr":"172.21.15.133:6823/14636","heartbeat_front_addr":"172.21.15.133:6821/14636","state":["exists","up"]},{"osd":2,"uuid":"1b98a02b-858a-4100-8da2-2021ed431999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":14637},{"type":"v1","addr":"172.21.15.133:6801","nonce":14637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":14637},{"type":"v1","addr":"172.21.15.133:6803","nonce":14637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":14637},{"type":"v1","addr":"172.21.15.133:6807","nonce":14637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":14637},{"type":"v1","addr":"172.21.15.133:6805","nonce":14637}]},"public_addr":"172.21.15.133:6801/14637","cluster_addr":"172.21.15.133:6803/14637","heartbeat_back_addr":"172.21.15.133:6807/14637","heartbeat_front_addr":"172.21.15.133:6805/14637","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:45:26.337258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:45:26.374686+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:45:26.018929+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:45:31.449 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-11T09:45:28.093603+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-02-11T09:45:29.100223+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-11T09:45:31.449 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get device_health_metrics pg_num 2021-02-11T09:45:31.748 INFO:teuthology.orchestra.run.smithi133.stdout:pg_num: 1 2021-02-11T09:45:31.768 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get rbd pg_num 2021-02-11T09:45:32.068 INFO:teuthology.orchestra.run.smithi133.stdout:pg_num: 8 2021-02-11T09:45:32.083 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2021-02-11T09:45:32.083 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2021-02-11T09:45:32.083 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2021-02-11T09:45:32.423 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:45:32.440 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":14452},{"type":"v1","addr":"172.21.15.133:6825","nonce":14452}]},"active_addr":"172.21.15.133:6825/14452","active_change":"2021-02-11T09:45:25.417051+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":1557854831}]},{"addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":1194359855}]}]}} 2021-02-11T09:45:32.442 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2021-02-11T09:45:32.442 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2021-02-11T09:45:32.442 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-02-11T09:45:32.737 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:45:32.737 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":13,"fsid":"4cba45bb-28f9-46d3-a27e-3c23d56a54ef","created":"2021-02-11T09:45:20.484106+0000","modified":"2021-02-11T09:45:32.113356+0000","last_up_change":"2021-02-11T09:45:28.077370+0000","last_in_change":"2021-02-11T09:45:28.077370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:45:28.093603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T09:45:29.100223+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"1bd20326-5d00-4271-8124-7670b3b725ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":14635},{"type":"v1","addr":"172.21.15.133:6809","nonce":14635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":14635},{"type":"v1","addr":"172.21.15.133:6811","nonce":14635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":14635},{"type":"v1","addr":"172.21.15.133:6815","nonce":14635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":14635},{"type":"v1","addr":"172.21.15.133:6813","nonce":14635}]},"public_addr":"172.21.15.133:6809/14635","cluster_addr":"172.21.15.133:6811/14635","heartbeat_back_addr":"172.21.15.133:6815/14635","heartbeat_front_addr":"172.21.15.133:6813/14635","state":["exists","up"]},{"osd":1,"uuid":"2e90ab89-8f3b-459a-96ff-00d2d91bac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":14636},{"type":"v1","addr":"172.21.15.133:6817","nonce":14636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":14636},{"type":"v1","addr":"172.21.15.133:6819","nonce":14636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":14636},{"type":"v1","addr":"172.21.15.133:6823","nonce":14636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":14636},{"type":"v1","addr":"172.21.15.133:6821","nonce":14636}]},"public_addr":"172.21.15.133:6817/14636","cluster_addr":"172.21.15.133:6819/14636","heartbeat_back_addr":"172.21.15.133:6823/14636","heartbeat_front_addr":"172.21.15.133:6821/14636","state":["exists","up"]},{"osd":2,"uuid":"1b98a02b-858a-4100-8da2-2021ed431999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":14637},{"type":"v1","addr":"172.21.15.133:6801","nonce":14637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":14637},{"type":"v1","addr":"172.21.15.133:6803","nonce":14637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":14637},{"type":"v1","addr":"172.21.15.133:6807","nonce":14637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":14637},{"type":"v1","addr":"172.21.15.133:6805","nonce":14637}]},"public_addr":"172.21.15.133:6801/14637","cluster_addr":"172.21.15.133:6803/14637","heartbeat_back_addr":"172.21.15.133:6807/14637","heartbeat_front_addr":"172.21.15.133:6805/14637","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:45:26.337258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:45:26.374686+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:45:26.018929+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:45:32.751 INFO:tasks.ceph.ceph_manager.c2:all up! 2021-02-11T09:45:32.751 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-02-11T09:45:33.042 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:45:33.042 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":13,"fsid":"4cba45bb-28f9-46d3-a27e-3c23d56a54ef","created":"2021-02-11T09:45:20.484106+0000","modified":"2021-02-11T09:45:32.113356+0000","last_up_change":"2021-02-11T09:45:28.077370+0000","last_in_change":"2021-02-11T09:45:28.077370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:45:28.093603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T09:45:29.100223+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"1bd20326-5d00-4271-8124-7670b3b725ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":14635},{"type":"v1","addr":"172.21.15.133:6809","nonce":14635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":14635},{"type":"v1","addr":"172.21.15.133:6811","nonce":14635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":14635},{"type":"v1","addr":"172.21.15.133:6815","nonce":14635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":14635},{"type":"v1","addr":"172.21.15.133:6813","nonce":14635}]},"public_addr":"172.21.15.133:6809/14635","cluster_addr":"172.21.15.133:6811/14635","heartbeat_back_addr":"172.21.15.133:6815/14635","heartbeat_front_addr":"172.21.15.133:6813/14635","state":["exists","up"]},{"osd":1,"uuid":"2e90ab89-8f3b-459a-96ff-00d2d91bac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":14636},{"type":"v1","addr":"172.21.15.133:6817","nonce":14636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":14636},{"type":"v1","addr":"172.21.15.133:6819","nonce":14636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":14636},{"type":"v1","addr":"172.21.15.133:6823","nonce":14636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":14636},{"type":"v1","addr":"172.21.15.133:6821","nonce":14636}]},"public_addr":"172.21.15.133:6817/14636","cluster_addr":"172.21.15.133:6819/14636","heartbeat_back_addr":"172.21.15.133:6823/14636","heartbeat_front_addr":"172.21.15.133:6821/14636","state":["exists","up"]},{"osd":2,"uuid":"1b98a02b-858a-4100-8da2-2021ed431999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":14637},{"type":"v1","addr":"172.21.15.133:6801","nonce":14637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":14637},{"type":"v1","addr":"172.21.15.133:6803","nonce":14637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":14637},{"type":"v1","addr":"172.21.15.133:6807","nonce":14637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":14637},{"type":"v1","addr":"172.21.15.133:6805","nonce":14637}]},"public_addr":"172.21.15.133:6801/14637","cluster_addr":"172.21.15.133:6803/14637","heartbeat_back_addr":"172.21.15.133:6807/14637","heartbeat_front_addr":"172.21.15.133:6805/14637","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:45:26.337258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:45:26.374686+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:45:26.018929+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:45:33.061 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2021-02-11T09:45:33.225 INFO:teuthology.orchestra.run.smithi133.stdout:38654705668 2021-02-11T09:45:33.225 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2021-02-11T09:45:33.387 INFO:teuthology.orchestra.run.smithi133.stdout:38654705668 2021-02-11T09:45:33.387 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2021-02-11T09:45:33.554 INFO:teuthology.orchestra.run.smithi133.stdout:34359738371 2021-02-11T09:45:33.554 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-02-11T09:45:33.852 INFO:teuthology.orchestra.run.smithi133.stdout:38654705667 2021-02-11T09:45:33.867 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705667 for osd.0 2021-02-11T09:45:34.868 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-02-11T09:45:35.171 INFO:teuthology.orchestra.run.smithi133.stdout:38654705668 2021-02-11T09:45:35.185 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705668 for osd.0 2021-02-11T09:45:35.185 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2021-02-11T09:45:35.482 INFO:teuthology.orchestra.run.smithi133.stdout:38654705668 2021-02-11T09:45:35.497 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705668 for osd.1 2021-02-11T09:45:35.497 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2021-02-11T09:45:35.794 INFO:teuthology.orchestra.run.smithi133.stdout:34359738370 2021-02-11T09:45:35.809 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.2 2021-02-11T09:45:36.810 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2021-02-11T09:45:37.118 INFO:teuthology.orchestra.run.smithi133.stdout:34359738371 2021-02-11T09:45:37.137 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.2 2021-02-11T09:45:37.137 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2021-02-11T09:45:37.137 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-02-11T09:45:37.423 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:45:37.425 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2021-02-11T09:45:37.441 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2021-02-11T09:45:35.426743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14696,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":14336,"kb_avail":314558104,"statfs":{"total":322122547200,"available":322107498496,"internally_reserved":0,"allocated":344064,"data_stored":95613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14680064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T09:45:31.130159+0000","last_change":"2021-02-11T09:45:31.128209+0000","last_active":"2021-02-11T09:45:31.130159+0000","last_peered":"2021-02-11T09:45:31.130159+0000","last_clean":"2021-02-11T09:45:31.130159+0000","last_became_active":"2021-02-11T09:45:31.127852+0000","last_became_peered":"2021-02-11T09:45:31.127852+0000","last_unstale":"2021-02-11T09:45:31.130159+0000","last_undegraded":"2021-02-11T09:45:31.130159+0000","last_fullsized":"2021-02-11T09:45:31.130159+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_clean_scrub_stamp":"2021-02-11T09:45:30.092971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T09:45:31.132403+0000","last_change":"2021-02-11T09:45:31.131175+0000","last_active":"2021-02-11T09:45:31.132403+0000","last_peered":"2021-02-11T09:45:31.132403+0000","last_clean":"2021-02-11T09:45:31.132403+0000","last_became_active":"2021-02-11T09:45:31.130855+0000","last_became_peered":"2021-02-11T09:45:31.130855+0000","last_unstale":"2021-02-11T09:45:31.132403+0000","last_undegraded":"2021-02-11T09:45:31.132403+0000","last_fullsized":"2021-02-11T09:45:31.132403+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_clean_scrub_stamp":"2021-02-11T09:45:30.092971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T09:45:31.130068+0000","last_change":"2021-02-11T09:45:31.128322+0000","last_active":"2021-02-11T09:45:31.130068+0000","last_peered":"2021-02-11T09:45:31.130068+0000","last_clean":"2021-02-11T09:45:31.130068+0000","last_became_active":"2021-02-11T09:45:31.127515+0000","last_became_peered":"2021-02-11T09:45:31.127515+0000","last_unstale":"2021-02-11T09:45:31.130068+0000","last_undegraded":"2021-02-11T09:45:31.130068+0000","last_fullsized":"2021-02-11T09:45:31.130068+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_clean_scrub_stamp":"2021-02-11T09:45:30.092971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T09:45:31.129444+0000","last_change":"2021-02-11T09:45:31.127980+0000","last_active":"2021-02-11T09:45:31.129444+0000","last_peered":"2021-02-11T09:45:31.129444+0000","last_clean":"2021-02-11T09:45:31.129444+0000","last_became_active":"2021-02-11T09:45:31.127261+0000","last_became_peered":"2021-02-11T09:45:31.127261+0000","last_unstale":"2021-02-11T09:45:31.129444+0000","last_undegraded":"2021-02-11T09:45:31.129444+0000","last_fullsized":"2021-02-11T09:45:31.129444+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_clean_scrub_stamp":"2021-02-11T09:45:30.092971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T09:45:31.124804+0000","last_change":"2021-02-11T09:45:30.114279+0000","last_active":"2021-02-11T09:45:31.124804+0000","last_peered":"2021-02-11T09:45:31.124804+0000","last_clean":"2021-02-11T09:45:31.124804+0000","last_became_active":"2021-02-11T09:45:30.113076+0000","last_became_peered":"2021-02-11T09:45:30.113076+0000","last_unstale":"2021-02-11T09:45:31.124804+0000","last_undegraded":"2021-02-11T09:45:31.124804+0000","last_fullsized":"2021-02-11T09:45:31.124804+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:45:29.086765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:45:29.086765+0000","last_clean_scrub_stamp":"2021-02-11T09:45:29.086765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T09:45:31.131696+0000","last_change":"2021-02-11T09:45:31.130249+0000","last_active":"2021-02-11T09:45:31.131696+0000","last_peered":"2021-02-11T09:45:31.131696+0000","last_clean":"2021-02-11T09:45:31.131696+0000","last_became_active":"2021-02-11T09:45:31.129214+0000","last_became_peered":"2021-02-11T09:45:31.129214+0000","last_unstale":"2021-02-11T09:45:31.131696+0000","last_undegraded":"2021-02-11T09:45:31.131696+0000","last_fullsized":"2021-02-11T09:45:31.131696+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_clean_scrub_stamp":"2021-02-11T09:45:30.092971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T09:45:31.129224+0000","last_change":"2021-02-11T09:45:31.127017+0000","last_active":"2021-02-11T09:45:31.129224+0000","last_peered":"2021-02-11T09:45:31.129224+0000","last_clean":"2021-02-11T09:45:31.129224+0000","last_became_active":"2021-02-11T09:45:31.126647+0000","last_became_peered":"2021-02-11T09:45:31.126647+0000","last_unstale":"2021-02-11T09:45:31.129224+0000","last_undegraded":"2021-02-11T09:45:31.129224+0000","last_fullsized":"2021-02-11T09:45:31.129224+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_clean_scrub_stamp":"2021-02-11T09:45:30.092971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T09:45:31.128783+0000","last_change":"2021-02-11T09:45:31.127852+0000","last_active":"2021-02-11T09:45:31.128783+0000","last_peered":"2021-02-11T09:45:31.128783+0000","last_clean":"2021-02-11T09:45:31.128783+0000","last_became_active":"2021-02-11T09:45:31.127536+0000","last_became_peered":"2021-02-11T09:45:31.127536+0000","last_unstale":"2021-02-11T09:45:31.128783+0000","last_undegraded":"2021-02-11T09:45:31.128783+0000","last_fullsized":"2021-02-11T09:45:31.128783+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_clean_scrub_stamp":"2021-02-11T09:45:30.092971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T09:45:31.131734+0000","last_change":"2021-02-11T09:45:31.130735+0000","last_active":"2021-02-11T09:45:31.131734+0000","last_peered":"2021-02-11T09:45:31.131734+0000","last_clean":"2021-02-11T09:45:31.131734+0000","last_became_active":"2021-02-11T09:45:31.130152+0000","last_became_peered":"2021-02-11T09:45:31.130152+0000","last_unstale":"2021-02-11T09:45:31.131734+0000","last_undegraded":"2021-02-11T09:45:31.131734+0000","last_fullsized":"2021-02-11T09:45:31.131734+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_clean_scrub_stamp":"2021-02-11T09:45:30.092971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":31871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.214}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6679999999999999}]}]},{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4856,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":104852744,"statfs":{"total":107374182400,"available":107369209856,"internally_reserved":0,"allocated":114688,"data_stored":31871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":31871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.837}]}]}]}} 2021-02-11T09:45:37.442 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-02-11T09:45:37.729 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:45:37.732 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2021-02-11T09:45:37.744 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-02-11T09:45:37.427273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14696,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":14336,"kb_avail":314558104,"statfs":{"total":322122547200,"available":322107498496,"internally_reserved":0,"allocated":344064,"data_stored":95613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14680064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T09:45:31.130159+0000","last_change":"2021-02-11T09:45:31.128209+0000","last_active":"2021-02-11T09:45:31.130159+0000","last_peered":"2021-02-11T09:45:31.130159+0000","last_clean":"2021-02-11T09:45:31.130159+0000","last_became_active":"2021-02-11T09:45:31.127852+0000","last_became_peered":"2021-02-11T09:45:31.127852+0000","last_unstale":"2021-02-11T09:45:31.130159+0000","last_undegraded":"2021-02-11T09:45:31.130159+0000","last_fullsized":"2021-02-11T09:45:31.130159+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_clean_scrub_stamp":"2021-02-11T09:45:30.092971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T09:45:31.132403+0000","last_change":"2021-02-11T09:45:31.131175+0000","last_active":"2021-02-11T09:45:31.132403+0000","last_peered":"2021-02-11T09:45:31.132403+0000","last_clean":"2021-02-11T09:45:31.132403+0000","last_became_active":"2021-02-11T09:45:31.130855+0000","last_became_peered":"2021-02-11T09:45:31.130855+0000","last_unstale":"2021-02-11T09:45:31.132403+0000","last_undegraded":"2021-02-11T09:45:31.132403+0000","last_fullsized":"2021-02-11T09:45:31.132403+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_clean_scrub_stamp":"2021-02-11T09:45:30.092971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T09:45:31.130068+0000","last_change":"2021-02-11T09:45:31.128322+0000","last_active":"2021-02-11T09:45:31.130068+0000","last_peered":"2021-02-11T09:45:31.130068+0000","last_clean":"2021-02-11T09:45:31.130068+0000","last_became_active":"2021-02-11T09:45:31.127515+0000","last_became_peered":"2021-02-11T09:45:31.127515+0000","last_unstale":"2021-02-11T09:45:31.130068+0000","last_undegraded":"2021-02-11T09:45:31.130068+0000","last_fullsized":"2021-02-11T09:45:31.130068+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_clean_scrub_stamp":"2021-02-11T09:45:30.092971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T09:45:31.129444+0000","last_change":"2021-02-11T09:45:31.127980+0000","last_active":"2021-02-11T09:45:31.129444+0000","last_peered":"2021-02-11T09:45:31.129444+0000","last_clean":"2021-02-11T09:45:31.129444+0000","last_became_active":"2021-02-11T09:45:31.127261+0000","last_became_peered":"2021-02-11T09:45:31.127261+0000","last_unstale":"2021-02-11T09:45:31.129444+0000","last_undegraded":"2021-02-11T09:45:31.129444+0000","last_fullsized":"2021-02-11T09:45:31.129444+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_clean_scrub_stamp":"2021-02-11T09:45:30.092971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T09:45:31.124804+0000","last_change":"2021-02-11T09:45:30.114279+0000","last_active":"2021-02-11T09:45:31.124804+0000","last_peered":"2021-02-11T09:45:31.124804+0000","last_clean":"2021-02-11T09:45:31.124804+0000","last_became_active":"2021-02-11T09:45:30.113076+0000","last_became_peered":"2021-02-11T09:45:30.113076+0000","last_unstale":"2021-02-11T09:45:31.124804+0000","last_undegraded":"2021-02-11T09:45:31.124804+0000","last_fullsized":"2021-02-11T09:45:31.124804+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:45:29.086765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:45:29.086765+0000","last_clean_scrub_stamp":"2021-02-11T09:45:29.086765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T09:45:31.131696+0000","last_change":"2021-02-11T09:45:31.130249+0000","last_active":"2021-02-11T09:45:31.131696+0000","last_peered":"2021-02-11T09:45:31.131696+0000","last_clean":"2021-02-11T09:45:31.131696+0000","last_became_active":"2021-02-11T09:45:31.129214+0000","last_became_peered":"2021-02-11T09:45:31.129214+0000","last_unstale":"2021-02-11T09:45:31.131696+0000","last_undegraded":"2021-02-11T09:45:31.131696+0000","last_fullsized":"2021-02-11T09:45:31.131696+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_clean_scrub_stamp":"2021-02-11T09:45:30.092971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T09:45:31.129224+0000","last_change":"2021-02-11T09:45:31.127017+0000","last_active":"2021-02-11T09:45:31.129224+0000","last_peered":"2021-02-11T09:45:31.129224+0000","last_clean":"2021-02-11T09:45:31.129224+0000","last_became_active":"2021-02-11T09:45:31.126647+0000","last_became_peered":"2021-02-11T09:45:31.126647+0000","last_unstale":"2021-02-11T09:45:31.129224+0000","last_undegraded":"2021-02-11T09:45:31.129224+0000","last_fullsized":"2021-02-11T09:45:31.129224+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_clean_scrub_stamp":"2021-02-11T09:45:30.092971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T09:45:31.128783+0000","last_change":"2021-02-11T09:45:31.127852+0000","last_active":"2021-02-11T09:45:31.128783+0000","last_peered":"2021-02-11T09:45:31.128783+0000","last_clean":"2021-02-11T09:45:31.128783+0000","last_became_active":"2021-02-11T09:45:31.127536+0000","last_became_peered":"2021-02-11T09:45:31.127536+0000","last_unstale":"2021-02-11T09:45:31.128783+0000","last_undegraded":"2021-02-11T09:45:31.128783+0000","last_fullsized":"2021-02-11T09:45:31.128783+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_clean_scrub_stamp":"2021-02-11T09:45:30.092971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T09:45:31.131734+0000","last_change":"2021-02-11T09:45:31.130735+0000","last_active":"2021-02-11T09:45:31.131734+0000","last_peered":"2021-02-11T09:45:31.131734+0000","last_clean":"2021-02-11T09:45:31.131734+0000","last_became_active":"2021-02-11T09:45:31.130152+0000","last_became_peered":"2021-02-11T09:45:31.130152+0000","last_unstale":"2021-02-11T09:45:31.131734+0000","last_undegraded":"2021-02-11T09:45:31.131734+0000","last_fullsized":"2021-02-11T09:45:31.131734+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:45:30.092971+0000","last_clean_scrub_stamp":"2021-02-11T09:45:30.092971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":31871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.214}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6679999999999999}]}]},{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4856,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":104852744,"statfs":{"total":107374182400,"available":107369209856,"internally_reserved":0,"allocated":114688,"data_stored":31871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":31871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.837}]}]}]}} 2021-02-11T09:45:37.745 INFO:tasks.ceph.ceph_manager.c2:clean! 2021-02-11T09:45:37.745 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2021-02-11T09:45:37.745 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2021-02-11T09:45:37.745 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2021-02-11T09:45:38.097 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T09:45:38.112 INFO:teuthology.orchestra.run.smithi133.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-02-11T09:45:38.112 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2021-02-11T09:45:38.112 INFO:teuthology.run_tasks:Running task rgw... 2021-02-11T09:45:38.123 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2021-02-11T09:45:38.123 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2021-02-11T09:45:38.123 INFO:tasks.rgw:Creating data pools 2021-02-11T09:45:38.124 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2021-02-11T09:45:38.124 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-02-11T09:45:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'default.rgw.buckets.data' created 2021-02-11T09:45:40.021 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-02-11T09:45:40.689 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-11T09:45:40.706 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-02-11T09:45:41.717 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'default.rgw.buckets.index' created 2021-02-11T09:45:41.734 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-02-11T09:45:42.725 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-11T09:45:42.747 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2021-02-11T09:45:42.748 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-02-11T09:45:43.044 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'default.rgw.buckets.data' already exists 2021-02-11T09:45:43.056 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-02-11T09:45:43.716 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-11T09:45:43.729 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-02-11T09:45:44.030 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'default.rgw.buckets.index' already exists 2021-02-11T09:45:44.041 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-02-11T09:45:44.726 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-11T09:45:44.739 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2021-02-11T09:45:44.739 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-02-11T09:45:45.509 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'default.rgw.buckets.data' created 2021-02-11T09:45:45.525 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-02-11T09:45:46.527 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-11T09:45:46.550 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-02-11T09:45:47.528 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'default.rgw.buckets.index' created 2021-02-11T09:45:47.543 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-02-11T09:45:48.541 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-11T09:45:48.559 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2021-02-11T09:45:48.559 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-02-11T09:45:48.843 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'default.rgw.buckets.data' already exists 2021-02-11T09:45:48.855 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-02-11T09:45:49.532 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-11T09:45:49.547 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-02-11T09:45:49.841 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'default.rgw.buckets.index' already exists 2021-02-11T09:45:49.852 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-02-11T09:45:50.541 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-11T09:45:50.554 DEBUG:tasks.rgw:Pools created 2021-02-11T09:45:50.554 INFO:tasks.rgw:Configuring compression type = random 2021-02-11T09:45:50.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2021-02-11T09:45:50.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2021-02-11T09:45:50.555 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2021-02-11T09:45:50.592 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:45:50.592 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:45:50.604 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:50.600+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:51.678 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.676+0000 7f21b9a00080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:45:51.678 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.676+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:51.685 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.684+0000 7f21b9a00080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:45:51.686 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.684+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:51.688 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.684+0000 7f21b9a00080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:45:51.688 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.684+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:51.688 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.684+0000 7f21b9a00080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:45:51.705 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.700+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:51.705 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.704+0000 7f21b9a00080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:45:51.705 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.704+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:51.706 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.704+0000 7f21b9a00080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:45:51.706 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.704+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:51.706 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.704+0000 7f21b9a00080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:45:51.706 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.704+0000 7f21b9a00080 10 cannot find current period zonegroup using local zonegroup 2021-02-11T09:45:51.706 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.704+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:51.706 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.704+0000 7f21b9a00080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:45:51.706 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.704+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:51.707 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.704+0000 7f21b9a00080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:45:51.707 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.704+0000 7f21b9a00080 10 Creating default zonegroup 2021-02-11T09:45:51.707 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.704+0000 7f21b9a00080 20 get_system_obj_state: rctx=0x7fffedc72e40 obj=:.pools.avail state=0x55876acb41f0 s->prefetch_data=0 2021-02-11T09:45:51.707 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.704+0000 7f21b9a00080 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-02-11T09:45:51.721 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.720+0000 7f21b9a00080 10 WARNING: store->list_zones() returned r=-2 2021-02-11T09:45:51.722 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.720+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:51.722 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.720+0000 7f21b9a00080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:45:51.727 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.724+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:51.727 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.724+0000 7f21b9a00080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:45:51.727 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.724+0000 7f21b9a00080 10 could not read realm id: (2) No such file or directory 2021-02-11T09:45:51.728 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.724+0000 7f21b9a00080 10 WARNING: failed to set zone as default, r=-22 2021-02-11T09:45:51.728 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.724+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:51.728 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.724+0000 7f21b9a00080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:45:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.732+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.732+0000 7f21b9a00080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T09:45:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.732+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.732+0000 7f21b9a00080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:45:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.732+0000 7f21b9a00080 20 zonegroup default 2021-02-11T09:45:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.732+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.732+0000 7f21b9a00080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:45:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.732+0000 7f21b9a00080 10 Cannot find current period zone using local zone 2021-02-11T09:45:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.732+0000 7f21b9a00080 10 Using default name default 2021-02-11T09:45:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.732+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.732+0000 7f21b9a00080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:45:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.732+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.732+0000 7f21b9a00080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T09:45:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.732+0000 7f21b9a00080 20 zone default found 2021-02-11T09:45:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:51.732+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:53.694 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:53.692+0000 7f21b9a00080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:45:53.695 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:53.692+0000 7f21b9a00080 20 rados->read ofs=0 len=0 2021-02-11T09:45:53.697 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:53.696+0000 7f21b9a00080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:45:53.697 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:53.696+0000 7f21b9a00080 20 started sync module instance, tier type = 2021-02-11T09:45:53.697 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:53.696+0000 7f21b9a00080 20 started zone id=78c64d11-4cbc-452c-a005-aed25e955c00 (name=default) with tier type = 2021-02-11T09:45:55.737 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:55.736+0000 7f21b9a00080 20 add_watcher() i=0 2021-02-11T09:45:55.738 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:55.736+0000 7f21b9a00080 20 add_watcher() i=1 2021-02-11T09:45:55.738 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:55.736+0000 7f21b9a00080 20 add_watcher() i=2 2021-02-11T09:45:55.738 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:55.736+0000 7f21b9a00080 20 add_watcher() i=3 2021-02-11T09:45:55.738 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:55.736+0000 7f21b9a00080 20 add_watcher() i=4 2021-02-11T09:45:55.738 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:55.736+0000 7f21b9a00080 20 add_watcher() i=5 2021-02-11T09:45:55.738 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:55.736+0000 7f21b9a00080 20 add_watcher() i=6 2021-02-11T09:45:55.738 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:55.736+0000 7f21b9a00080 20 add_watcher() i=7 2021-02-11T09:45:55.739 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:55.736+0000 7f21b9a00080 2 all 8 watchers are set, enabling cache 2021-02-11T09:45:55.739 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:55.736+0000 7f21b9a00080 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(optional_yield):423: oid=data_loggenerations_metadata not found 2021-02-11T09:45:55.745 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:55.744+0000 7f21767e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T09:45:55.745 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:55.744+0000 7f21b9a00080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T09:45:55.745 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:55.744+0000 7f21b9a00080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T09:45:59.704 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.700+0000 7f21b9a00080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-02-11T09:45:59.820 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.816+0000 7f20e17fa700 20 reqs_thread_entry: start 2021-02-11T09:45:59.820 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.816+0000 7f20d2ffd700 20 reqs_thread_entry: start 2021-02-11T09:45:59.820 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.816+0000 7f21b9a00080 20 init_complete bucket index max shards: 11 2021-02-11T09:45:59.820 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.816+0000 7f21b9a00080 10 rgw notify: Started notification manager with: 1 workers 2021-02-11T09:45:59.820 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.816+0000 7f20d17fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Feb 11 09:46:30 2021 2021-02-11T09:45:59.820 INFO:teuthology.orchestra.run.smithi026.stderr: 2021-02-11T09:45:59.821 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.816+0000 7f20d0ff9700 20 reqs_thread_entry: start 2021-02-11T09:45:59.854 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.852+0000 7f21b9a00080 20 remove_watcher() i=0 2021-02-11T09:45:59.854 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.852+0000 7f21b9a00080 2 removed watcher, disabling cache 2021-02-11T09:45:59.857 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.856+0000 7f21b9a00080 20 remove_watcher() i=1 2021-02-11T09:45:59.860 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.856+0000 7f21b9a00080 20 remove_watcher() i=2 2021-02-11T09:45:59.863 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.860+0000 7f21b9a00080 20 remove_watcher() i=3 2021-02-11T09:45:59.866 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.864+0000 7f21b9a00080 20 remove_watcher() i=4 2021-02-11T09:45:59.870 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.868+0000 7f21b9a00080 20 remove_watcher() i=5 2021-02-11T09:45:59.872 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.868+0000 7f21b9a00080 20 remove_watcher() i=6 2021-02-11T09:45:59.875 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.872+0000 7f21b9a00080 20 remove_watcher() i=7 2021-02-11T09:45:59.877 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.876+0000 7f21767e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T09:45:59.886 INFO:teuthology.orchestra.run.smithi026.stdout:[] 2021-02-11T09:45:59.887 DEBUG:tasks.util.rgw: json result: [] 2021-02-11T09:45:59.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T09:45:59.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T09:45:59.887 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-11T09:45:59.969 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:45:59.969 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:45:59.981 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.980+0000 7fbfd3683080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T09:45:59.982 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.980+0000 7fbfd3683080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T09:45:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.988+0000 7fbf07fff700 20 reqs_thread_entry: start 2021-02-11T09:45:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.988+0000 7fbfd3683080 20 rados->read ofs=0 len=0 2021-02-11T09:45:59.992 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.988+0000 7fbfd3683080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:45:59.992 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.988+0000 7fbfd3683080 20 rados->read ofs=0 len=0 2021-02-11T09:45:59.993 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.992+0000 7fbfd3683080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T09:45:59.993 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.992+0000 7fbfd3683080 20 rados->read ofs=0 len=0 2021-02-11T09:45:59.993 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.992+0000 7fbfd3683080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:45:59.993 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.992+0000 7fbfd3683080 20 rados->read ofs=0 len=0 2021-02-11T09:45:59.994 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.992+0000 7fbfd3683080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:45:59.994 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.992+0000 7fbfd3683080 20 rados->read ofs=0 len=0 2021-02-11T09:45:59.995 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:45:59.992+0000 7fbfd3683080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:46:00.006 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"78c64d11-4cbc-452c-a005-aed25e955c00","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2021-02-11T09:46:00.006 DEBUG:tasks.util.rgw: json result: {'id': '78c64d11-4cbc-452c-a005-aed25e955c00', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-02-11T09:46:00.006 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2021-02-11T09:46:00.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2021-02-11T09:46:00.006 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2021-02-11T09:46:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:00.085 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.092+0000 7fb8211b0080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.092+0000 7fb8211b0080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.092+0000 7fb8211b0080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.092+0000 7fb8211b0080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.092+0000 7fb8211b0080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.092+0000 7fb8211b0080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.092+0000 7fb8211b0080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.092+0000 7fb8211b0080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:46:00.116 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.112+0000 7fb8211b0080 20 RGWRados::pool_iterate: got zone_info.78c64d11-4cbc-452c-a005-aed25e955c00 2021-02-11T09:46:00.116 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.112+0000 7fb8211b0080 20 RGWRados::pool_iterate: got zonegroup_info.13d0d618-fada-44ca-bb4c-3f624affc233 2021-02-11T09:46:00.116 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.112+0000 7fb8211b0080 20 RGWRados::pool_iterate: got zone_names.default 2021-02-11T09:46:00.116 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.112+0000 7fb8211b0080 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-02-11T09:46:00.116 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.112+0000 7fb8211b0080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.117 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.112+0000 7fb8211b0080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:46:00.117 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.112+0000 7fb8211b0080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.118 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.116+0000 7fb8211b0080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T09:46:00.118 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.116+0000 7fb8211b0080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.118 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.116+0000 7fb8211b0080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:46:00.118 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.116+0000 7fb8211b0080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.119 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.116+0000 7fb8211b0080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:46:00.119 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.116+0000 7fb8211b0080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.119 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.116+0000 7fb8211b0080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:00.119 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.116+0000 7fb8211b0080 10 cannot find current period zonegroup using local zonegroup 2021-02-11T09:46:00.120 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.116+0000 7fb8211b0080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.120 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.116+0000 7fb8211b0080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:00.120 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.116+0000 7fb8211b0080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.121 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.116+0000 7fb8211b0080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:46:00.121 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.116+0000 7fb8211b0080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.121 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.120+0000 7fb8211b0080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:46:00.121 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.120+0000 7fb8211b0080 20 zonegroup default 2021-02-11T09:46:00.122 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.120+0000 7fb8211b0080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.122 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.120+0000 7fb8211b0080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:00.122 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.120+0000 7fb8211b0080 10 Cannot find current period zone using local zone 2021-02-11T09:46:00.122 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.120+0000 7fb8211b0080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.123 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.120+0000 7fb8211b0080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:00.123 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.120+0000 7fb8211b0080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.123 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.120+0000 7fb8211b0080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:46:00.123 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.120+0000 7fb8211b0080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.124 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.120+0000 7fb8211b0080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T09:46:00.124 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.120+0000 7fb8211b0080 20 zone default found 2021-02-11T09:46:00.124 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.120+0000 7fb8211b0080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.124 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.120+0000 7fb8211b0080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:00.125 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.120+0000 7fb8211b0080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.125 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.124+0000 7fb8211b0080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:00.125 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.124+0000 7fb8211b0080 20 started sync module instance, tier type = 2021-02-11T09:46:00.125 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.124+0000 7fb8211b0080 20 started zone id=78c64d11-4cbc-452c-a005-aed25e955c00 (name=default) with tier type = 2021-02-11T09:46:00.145 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.144+0000 7fb8211b0080 20 add_watcher() i=0 2021-02-11T09:46:00.146 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.144+0000 7fb8211b0080 20 add_watcher() i=1 2021-02-11T09:46:00.146 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.144+0000 7fb8211b0080 20 add_watcher() i=2 2021-02-11T09:46:00.146 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.144+0000 7fb8211b0080 20 add_watcher() i=3 2021-02-11T09:46:00.146 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.144+0000 7fb8211b0080 20 add_watcher() i=4 2021-02-11T09:46:00.146 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.144+0000 7fb8211b0080 20 add_watcher() i=5 2021-02-11T09:46:00.146 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.144+0000 7fb8211b0080 20 add_watcher() i=6 2021-02-11T09:46:00.147 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.144+0000 7fb8211b0080 20 add_watcher() i=7 2021-02-11T09:46:00.147 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.144+0000 7fb8211b0080 2 all 8 watchers are set, enabling cache 2021-02-11T09:46:00.150 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.148+0000 7fb7de7e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T09:46:00.150 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.148+0000 7fb8211b0080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T09:46:00.150 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.148+0000 7fb8211b0080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T09:46:00.239 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.236+0000 7fb74d7fa700 20 reqs_thread_entry: start 2021-02-11T09:46:00.240 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.236+0000 7fb746ffd700 20 reqs_thread_entry: start 2021-02-11T09:46:00.240 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.236+0000 7fb8211b0080 20 init_complete bucket index max shards: 11 2021-02-11T09:46:00.240 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.236+0000 7fb8211b0080 10 rgw notify: Started notification manager with: 1 workers 2021-02-11T09:46:00.241 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.236+0000 7fb744ff9700 20 reqs_thread_entry: start 2021-02-11T09:46:00.241 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.236+0000 7fb7457fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Feb 11 09:46:30 2021 2021-02-11T09:46:00.241 INFO:teuthology.orchestra.run.smithi026.stderr: 2021-02-11T09:46:00.272 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.268+0000 7fb8211b0080 20 remove_watcher() i=0 2021-02-11T09:46:00.272 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.268+0000 7fb8211b0080 2 removed watcher, disabling cache 2021-02-11T09:46:00.275 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.272+0000 7fb8211b0080 20 remove_watcher() i=1 2021-02-11T09:46:00.278 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.276+0000 7fb8211b0080 20 remove_watcher() i=2 2021-02-11T09:46:00.281 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.280+0000 7fb8211b0080 20 remove_watcher() i=3 2021-02-11T09:46:00.284 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.280+0000 7fb8211b0080 20 remove_watcher() i=4 2021-02-11T09:46:00.287 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.284+0000 7fb8211b0080 20 remove_watcher() i=5 2021-02-11T09:46:00.290 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.288+0000 7fb8211b0080 20 remove_watcher() i=6 2021-02-11T09:46:00.293 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.292+0000 7fb8211b0080 20 remove_watcher() i=7 2021-02-11T09:46:00.295 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.292+0000 7fb7de7e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T09:46:00.305 INFO:teuthology.orchestra.run.smithi026.stdout:[] 2021-02-11T09:46:00.305 DEBUG:tasks.util.rgw: json result: [] 2021-02-11T09:46:00.305 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T09:46:00.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T09:46:00.305 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-11T09:46:00.390 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:00.390 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:00.401 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.400+0000 7f621661e080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T09:46:00.402 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.400+0000 7f621661e080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T09:46:00.407 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.404+0000 7f614bfff700 20 reqs_thread_entry: start 2021-02-11T09:46:00.407 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.404+0000 7f621661e080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.409 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.408+0000 7f621661e080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:46:00.409 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.408+0000 7f621661e080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.410 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.408+0000 7f621661e080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T09:46:00.410 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.408+0000 7f621661e080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.410 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.408+0000 7f621661e080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:00.410 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.408+0000 7f621661e080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.411 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.408+0000 7f621661e080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:46:00.411 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.408+0000 7f621661e080 20 rados->read ofs=0 len=0 2021-02-11T09:46:00.411 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:00.408+0000 7f621661e080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:46:00.421 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"78c64d11-4cbc-452c-a005-aed25e955c00","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2021-02-11T09:46:00.421 DEBUG:tasks.util.rgw: json result: {'id': '78c64d11-4cbc-452c-a005-aed25e955c00', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-02-11T09:46:00.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2021-02-11T09:46:00.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2021-02-11T09:46:00.421 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2021-02-11T09:46:00.457 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:00.457 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:00.470 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:00.470+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:02.492 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.490+0000 7f2d54e0b080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:02.493 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.490+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:02.506 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.506+0000 7f2d54e0b080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:02.506 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.506+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:02.507 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.506+0000 7f2d54e0b080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:02.508 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.506+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:02.509 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.506+0000 7f2d54e0b080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:02.533 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.530+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:02.533 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.530+0000 7f2d54e0b080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:02.533 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.530+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:02.533 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.530+0000 7f2d54e0b080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:02.534 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.530+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:02.534 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.534+0000 7f2d54e0b080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:02.534 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.534+0000 7f2d54e0b080 10 cannot find current period zonegroup using local zonegroup 2021-02-11T09:46:02.534 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.534+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:02.535 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.534+0000 7f2d54e0b080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:02.535 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.534+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:02.535 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.534+0000 7f2d54e0b080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:02.535 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.534+0000 7f2d54e0b080 10 Creating default zonegroup 2021-02-11T09:46:02.535 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.534+0000 7f2d54e0b080 20 get_system_obj_state: rctx=0x7ffe77975650 obj=:.pools.avail state=0x55c780c601f0 s->prefetch_data=0 2021-02-11T09:46:02.535 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.534+0000 7f2d54e0b080 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-02-11T09:46:02.550 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.550+0000 7f2d54e0b080 10 WARNING: store->list_zones() returned r=-2 2021-02-11T09:46:02.550 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.550+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:02.551 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.550+0000 7f2d54e0b080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:02.557 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.554+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:02.557 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.554+0000 7f2d54e0b080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:02.558 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.554+0000 7f2d54e0b080 10 could not read realm id: (2) No such file or directory 2021-02-11T09:46:02.558 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.554+0000 7f2d54e0b080 10 WARNING: failed to set zone as default, r=-22 2021-02-11T09:46:02.558 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.554+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:02.558 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.554+0000 7f2d54e0b080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:02.562 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.558+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:02.562 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.562+0000 7f2d54e0b080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T09:46:02.563 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.562+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:02.563 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.562+0000 7f2d54e0b080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:46:02.563 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.562+0000 7f2d54e0b080 20 zonegroup default 2021-02-11T09:46:02.563 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.562+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:02.564 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.562+0000 7f2d54e0b080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:02.564 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.562+0000 7f2d54e0b080 10 Cannot find current period zone using local zone 2021-02-11T09:46:02.564 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.562+0000 7f2d54e0b080 10 Using default name default 2021-02-11T09:46:02.564 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.562+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:02.565 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.562+0000 7f2d54e0b080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:46:02.565 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.562+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:02.565 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.562+0000 7f2d54e0b080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T09:46:02.565 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.562+0000 7f2d54e0b080 20 zone default found 2021-02-11T09:46:02.565 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:02.562+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:04.535 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:04.534+0000 7f2d54e0b080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:04.535 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:04.534+0000 7f2d54e0b080 20 rados->read ofs=0 len=0 2021-02-11T09:46:04.535 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:04.534+0000 7f2d54e0b080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:04.535 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:04.534+0000 7f2d54e0b080 20 started sync module instance, tier type = 2021-02-11T09:46:04.536 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:04.534+0000 7f2d54e0b080 20 started zone id=664b590b-0f37-4431-b0fd-81ccf4a30e41 (name=default) with tier type = 2021-02-11T09:46:06.576 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:06.574+0000 7f2d54e0b080 20 add_watcher() i=0 2021-02-11T09:46:06.577 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:06.574+0000 7f2d54e0b080 20 add_watcher() i=1 2021-02-11T09:46:06.577 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:06.574+0000 7f2d54e0b080 20 add_watcher() i=2 2021-02-11T09:46:06.577 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:06.574+0000 7f2d54e0b080 20 add_watcher() i=3 2021-02-11T09:46:06.577 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:06.574+0000 7f2d54e0b080 20 add_watcher() i=4 2021-02-11T09:46:06.577 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:06.574+0000 7f2d54e0b080 20 add_watcher() i=5 2021-02-11T09:46:06.578 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:06.574+0000 7f2d54e0b080 20 add_watcher() i=6 2021-02-11T09:46:06.578 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:06.574+0000 7f2d54e0b080 20 add_watcher() i=7 2021-02-11T09:46:06.578 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:06.574+0000 7f2d54e0b080 2 all 8 watchers are set, enabling cache 2021-02-11T09:46:06.579 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:06.578+0000 7f2d54e0b080 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(optional_yield):423: oid=data_loggenerations_metadata not found 2021-02-11T09:46:06.586 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:06.586+0000 7f2d11feb700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T09:46:06.586 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:06.586+0000 7f2d54e0b080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T09:46:06.587 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:06.586+0000 7f2d54e0b080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T09:46:10.538 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.534+0000 7f2d54e0b080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-02-11T09:46:10.652 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.650+0000 7f2c80ff9700 20 reqs_thread_entry: start 2021-02-11T09:46:10.652 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.650+0000 7f2c6a7fc700 20 reqs_thread_entry: start 2021-02-11T09:46:10.652 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.650+0000 7f2d54e0b080 20 init_complete bucket index max shards: 11 2021-02-11T09:46:10.652 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.650+0000 7f2d54e0b080 10 rgw notify: Started notification manager with: 1 workers 2021-02-11T09:46:10.652 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.650+0000 7f2c68ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Feb 11 09:46:40 2021 2021-02-11T09:46:10.653 INFO:teuthology.orchestra.run.smithi133.stderr: 2021-02-11T09:46:10.653 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.650+0000 7f2c53fff700 20 reqs_thread_entry: start 2021-02-11T09:46:10.678 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.678+0000 7f2d54e0b080 20 remove_watcher() i=0 2021-02-11T09:46:10.678 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.678+0000 7f2d54e0b080 2 removed watcher, disabling cache 2021-02-11T09:46:10.680 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.678+0000 7f2d54e0b080 20 remove_watcher() i=1 2021-02-11T09:46:10.685 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.682+0000 7f2d54e0b080 20 remove_watcher() i=2 2021-02-11T09:46:10.688 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.686+0000 7f2d54e0b080 20 remove_watcher() i=3 2021-02-11T09:46:10.691 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.690+0000 7f2d54e0b080 20 remove_watcher() i=4 2021-02-11T09:46:10.694 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.694+0000 7f2d54e0b080 20 remove_watcher() i=5 2021-02-11T09:46:10.697 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.694+0000 7f2d54e0b080 20 remove_watcher() i=6 2021-02-11T09:46:10.700 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.698+0000 7f2d54e0b080 20 remove_watcher() i=7 2021-02-11T09:46:10.701 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.698+0000 7f2d11feb700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T09:46:10.711 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2021-02-11T09:46:10.711 DEBUG:tasks.util.rgw: json result: [] 2021-02-11T09:46:10.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T09:46:10.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T09:46:10.712 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-11T09:46:10.797 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:10.797 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:10.806 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.806+0000 7fc2d31b4080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T09:46:10.807 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.806+0000 7fc2d31b4080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T09:46:10.814 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.810+0000 7fc207fff700 20 reqs_thread_entry: start 2021-02-11T09:46:10.814 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.810+0000 7fc2d31b4080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.816 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.814+0000 7fc2d31b4080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:46:10.816 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.814+0000 7fc2d31b4080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.817 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.814+0000 7fc2d31b4080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T09:46:10.817 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.814+0000 7fc2d31b4080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.818 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.818+0000 7fc2d31b4080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:10.818 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.818+0000 7fc2d31b4080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.819 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.818+0000 7fc2d31b4080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:46:10.819 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.818+0000 7fc2d31b4080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.821 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.818+0000 7fc2d31b4080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:46:10.831 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"664b590b-0f37-4431-b0fd-81ccf4a30e41","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2021-02-11T09:46:10.831 DEBUG:tasks.util.rgw: json result: {'id': '664b590b-0f37-4431-b0fd-81ccf4a30e41', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-02-11T09:46:10.831 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2021-02-11T09:46:10.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2021-02-11T09:46:10.831 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2021-02-11T09:46:10.908 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:10.908 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:10.919 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.918+0000 7f1f1d38d080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.923 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.922+0000 7f1f1d38d080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:10.923 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.922+0000 7f1f1d38d080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.923 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.922+0000 7f1f1d38d080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:10.923 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.922+0000 7f1f1d38d080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.924 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.922+0000 7f1f1d38d080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:10.924 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.922+0000 7f1f1d38d080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.924 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.922+0000 7f1f1d38d080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:46:10.948 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.946+0000 7f1f1d38d080 20 RGWRados::pool_iterate: got zonegroup_info.5c0f4268-4a7c-4a6b-90c8-f0de0bb05232 2021-02-11T09:46:10.948 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.946+0000 7f1f1d38d080 20 RGWRados::pool_iterate: got zone_info.664b590b-0f37-4431-b0fd-81ccf4a30e41 2021-02-11T09:46:10.948 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.946+0000 7f1f1d38d080 20 RGWRados::pool_iterate: got zone_names.default 2021-02-11T09:46:10.948 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.946+0000 7f1f1d38d080 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-02-11T09:46:10.948 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.946+0000 7f1f1d38d080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.949 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.946+0000 7f1f1d38d080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:46:10.949 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.946+0000 7f1f1d38d080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.950 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.946+0000 7f1f1d38d080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T09:46:10.950 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.946+0000 7f1f1d38d080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.950 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.950+0000 7f1f1d38d080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:46:10.950 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.950+0000 7f1f1d38d080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.951 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.950+0000 7f1f1d38d080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:46:10.951 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.950+0000 7f1f1d38d080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.952 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.950+0000 7f1f1d38d080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:10.952 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.950+0000 7f1f1d38d080 10 cannot find current period zonegroup using local zonegroup 2021-02-11T09:46:10.952 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.950+0000 7f1f1d38d080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.952 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.950+0000 7f1f1d38d080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:10.952 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.950+0000 7f1f1d38d080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.953 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.950+0000 7f1f1d38d080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:46:10.953 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.950+0000 7f1f1d38d080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.954 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.950+0000 7f1f1d38d080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:46:10.954 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.950+0000 7f1f1d38d080 20 zonegroup default 2021-02-11T09:46:10.954 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.950+0000 7f1f1d38d080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.955 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.954+0000 7f1f1d38d080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:10.955 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.954+0000 7f1f1d38d080 10 Cannot find current period zone using local zone 2021-02-11T09:46:10.955 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.954+0000 7f1f1d38d080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.956 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.954+0000 7f1f1d38d080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:10.956 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.954+0000 7f1f1d38d080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.956 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.954+0000 7f1f1d38d080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:46:10.956 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.954+0000 7f1f1d38d080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.957 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.954+0000 7f1f1d38d080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T09:46:10.957 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.954+0000 7f1f1d38d080 20 zone default found 2021-02-11T09:46:10.957 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.954+0000 7f1f1d38d080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.957 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.954+0000 7f1f1d38d080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:10.958 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.954+0000 7f1f1d38d080 20 rados->read ofs=0 len=0 2021-02-11T09:46:10.958 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.958+0000 7f1f1d38d080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:10.958 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.958+0000 7f1f1d38d080 20 started sync module instance, tier type = 2021-02-11T09:46:10.958 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.958+0000 7f1f1d38d080 20 started zone id=664b590b-0f37-4431-b0fd-81ccf4a30e41 (name=default) with tier type = 2021-02-11T09:46:10.982 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.982+0000 7f1f1d38d080 20 add_watcher() i=0 2021-02-11T09:46:10.982 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.982+0000 7f1f1d38d080 20 add_watcher() i=1 2021-02-11T09:46:10.982 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.982+0000 7f1f1d38d080 20 add_watcher() i=2 2021-02-11T09:46:10.982 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.982+0000 7f1f1d38d080 20 add_watcher() i=3 2021-02-11T09:46:10.982 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.982+0000 7f1f1d38d080 20 add_watcher() i=4 2021-02-11T09:46:10.983 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.982+0000 7f1f1d38d080 20 add_watcher() i=5 2021-02-11T09:46:10.983 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.982+0000 7f1f1d38d080 20 add_watcher() i=6 2021-02-11T09:46:10.985 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.982+0000 7f1f1d38d080 20 add_watcher() i=7 2021-02-11T09:46:10.985 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.982+0000 7f1f1d38d080 2 all 8 watchers are set, enabling cache 2021-02-11T09:46:10.990 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.990+0000 7f1eda7e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T09:46:10.990 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.990+0000 7f1f1d38d080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T09:46:10.990 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:10.990+0000 7f1f1d38d080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T09:46:11.093 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.090+0000 7f1e497fa700 20 reqs_thread_entry: start 2021-02-11T09:46:11.094 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.090+0000 7f1e42ffd700 20 reqs_thread_entry: start 2021-02-11T09:46:11.094 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.090+0000 7f1f1d38d080 20 init_complete bucket index max shards: 11 2021-02-11T09:46:11.094 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.090+0000 7f1f1d38d080 10 rgw notify: Started notification manager with: 1 workers 2021-02-11T09:46:11.094 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.090+0000 7f1e417fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Feb 11 09:46:41 2021 2021-02-11T09:46:11.094 INFO:teuthology.orchestra.run.smithi133.stderr: 2021-02-11T09:46:11.095 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.094+0000 7f1e40ff9700 20 reqs_thread_entry: start 2021-02-11T09:46:11.123 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.122+0000 7f1f1d38d080 20 remove_watcher() i=0 2021-02-11T09:46:11.124 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.122+0000 7f1f1d38d080 2 removed watcher, disabling cache 2021-02-11T09:46:11.126 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.126+0000 7f1f1d38d080 20 remove_watcher() i=1 2021-02-11T09:46:11.132 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.130+0000 7f1f1d38d080 20 remove_watcher() i=2 2021-02-11T09:46:11.135 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.134+0000 7f1f1d38d080 20 remove_watcher() i=3 2021-02-11T09:46:11.138 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.134+0000 7f1f1d38d080 20 remove_watcher() i=4 2021-02-11T09:46:11.141 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.138+0000 7f1f1d38d080 20 remove_watcher() i=5 2021-02-11T09:46:11.143 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.142+0000 7f1f1d38d080 20 remove_watcher() i=6 2021-02-11T09:46:11.146 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.146+0000 7f1f1d38d080 20 remove_watcher() i=7 2021-02-11T09:46:11.148 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.146+0000 7f1eda7e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T09:46:11.157 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2021-02-11T09:46:11.157 DEBUG:tasks.util.rgw: json result: [] 2021-02-11T09:46:11.158 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T09:46:11.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T09:46:11.158 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-11T09:46:11.241 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:11.241 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:11.254 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.250+0000 7f837b914080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T09:46:11.254 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.250+0000 7f837b914080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T09:46:11.262 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.258+0000 7f82affff700 20 reqs_thread_entry: start 2021-02-11T09:46:11.262 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.262+0000 7f837b914080 20 rados->read ofs=0 len=0 2021-02-11T09:46:11.264 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.262+0000 7f837b914080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:46:11.264 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.262+0000 7f837b914080 20 rados->read ofs=0 len=0 2021-02-11T09:46:11.266 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.262+0000 7f837b914080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T09:46:11.266 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.262+0000 7f837b914080 20 rados->read ofs=0 len=0 2021-02-11T09:46:11.266 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.266+0000 7f837b914080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:46:11.267 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.266+0000 7f837b914080 20 rados->read ofs=0 len=0 2021-02-11T09:46:11.267 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.266+0000 7f837b914080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:46:11.268 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.266+0000 7f837b914080 20 rados->read ofs=0 len=0 2021-02-11T09:46:11.269 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:11.266+0000 7f837b914080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:46:11.279 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"664b590b-0f37-4431-b0fd-81ccf4a30e41","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2021-02-11T09:46:11.279 DEBUG:tasks.util.rgw: json result: {'id': '664b590b-0f37-4431-b0fd-81ccf4a30e41', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-02-11T09:46:11.279 INFO:tasks.rgw:Starting rgw... 2021-02-11T09:46:11.279 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2021-02-11T09:46:11.280 INFO:tasks.rgw:Using civetweb as radosgw frontend 2021-02-11T09:46:11.280 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-02-11T09:46:11.280 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-02-11T09:46:11.282 INFO:tasks.rgw.c1.client.0:Started 2021-02-11T09:46:11.282 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2021-02-11T09:46:11.282 INFO:tasks.rgw:Using civetweb as radosgw frontend 2021-02-11T09:46:11.282 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-02-11T09:46:11.283 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-02-11T09:46:11.285 INFO:tasks.rgw.c1.client.1:Started 2021-02-11T09:46:11.285 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2021-02-11T09:46:11.285 INFO:tasks.rgw:Using civetweb as radosgw frontend 2021-02-11T09:46:11.285 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-02-11T09:46:11.285 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-02-11T09:46:11.324 INFO:tasks.rgw.c2.client.0:Started 2021-02-11T09:46:11.324 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2021-02-11T09:46:11.324 INFO:tasks.rgw:Using civetweb as radosgw frontend 2021-02-11T09:46:11.324 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-02-11T09:46:11.324 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-02-11T09:46:11.326 INFO:tasks.rgw.c2.client.1:Started 2021-02-11T09:46:11.326 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi026.front.sepia.ceph.com:8000/ 2021-02-11T09:46:11.326 DEBUG:teuthology.orchestra.run.smithi026:> curl http://smithi026.front.sepia.ceph.com:8000/ 2021-02-11T09:46:11.365 INFO:teuthology.orchestra.run.smithi026.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:46:11.365 INFO:teuthology.orchestra.run.smithi026.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:46:11.370 INFO:teuthology.orchestra.run.smithi026.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi026.front.sepia.ceph.com port 8000: Connection refused 2021-02-11T09:46:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T09:46:12.372 DEBUG:teuthology.orchestra.run.smithi026:> curl http://smithi026.front.sepia.ceph.com:8000/ 2021-02-11T09:46:12.388 INFO:teuthology.orchestra.run.smithi026.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:46:12.388 INFO:teuthology.orchestra.run.smithi026.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:46:12.394 INFO:teuthology.orchestra.run.smithi026.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2021-02-11T09:46:12.395 INFO:teuthology.orchestra.run.smithi026.stdout:anonymous 2021-02-11T09:46:12.395 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi026.front.sepia.ceph.com:8001/ 2021-02-11T09:46:12.395 DEBUG:teuthology.orchestra.run.smithi026:> curl http://smithi026.front.sepia.ceph.com:8001/ 2021-02-11T09:46:12.453 INFO:teuthology.orchestra.run.smithi026.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:46:12.453 INFO:teuthology.orchestra.run.smithi026.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:46:12.459 INFO:teuthology.orchestra.run.smithi026.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2021-02-11T09:46:12.460 INFO:teuthology.orchestra.run.smithi026.stdout:anonymous 2021-02-11T09:46:12.460 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi133.front.sepia.ceph.com:8000/ 2021-02-11T09:46:12.460 DEBUG:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8000/ 2021-02-11T09:46:12.682 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:46:12.682 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:46:12.689 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 17833 0 --:--:-- --:--:-- --:--:-- 35666 2021-02-11T09:46:12.689 INFO:teuthology.orchestra.run.smithi133.stdout:anonymous 2021-02-11T09:46:12.690 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi133.front.sepia.ceph.com:8001/ 2021-02-11T09:46:12.690 DEBUG:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8001/ 2021-02-11T09:46:12.707 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:46:12.707 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:46:12.714 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2021-02-11T09:46:12.715 INFO:teuthology.orchestra.run.smithi133.stdout:anonymous 2021-02-11T09:46:12.715 INFO:teuthology.run_tasks:Running task rgw-multisite... 2021-02-11T09:46:12.867 INFO:tasks.rgw_multisite:creating realm.. 2021-02-11T09:46:12.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:12.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:12.868 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:46:12.906 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:12.906 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:12.958 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"52cf47b0-3540-4b08-a664-7ef37f763895","name":"test-realm","current_period":"8ab84e61-9000-48cf-8393-6e16e4da90b0","epoch":1} 2021-02-11T09:46:12.959 INFO:tasks.rgw_multisite:creating master zonegroup.. 2021-02-11T09:46:12.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:12.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:12.959 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:46:13.040 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:13.040 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:13.061 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:13.060+0000 7fef44d3f080 -1 old format 2021-02-11T09:46:13.077 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","sync_policy":{"groups":[]}} 2021-02-11T09:46:13.078 INFO:tasks.rgw_multisite:creating master zone.. 2021-02-11T09:46:13.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'DHA6OWMQ42PGF1T7', '--secret', 'xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o', '--endpoints', 'http://smithi026.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:13.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'DHA6OWMQ42PGF1T7', '--secret', 'xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o', '--endpoints', 'http://smithi026.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:13.078 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key DHA6OWMQ42PGF1T7 --secret xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o --endpoints http://smithi026.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:46:13.168 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:13.168 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:13.249 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"a3c4910a-4529-4294-aed4-ef409f209cf5","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"DHA6OWMQ42PGF1T7","secret_key":"xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","notif_pool":"test-zone1.rgw.log:notif"} 2021-02-11T09:46:13.250 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2021-02-11T09:46:13.669 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'test-zone1.rgw.buckets.data' created 2021-02-11T09:46:13.692 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2021-02-11T09:46:14.709 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2021-02-11T09:46:14.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:14.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:14.731 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:46:14.770 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:14.770 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:14.801 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"a3c4910a-4529-4294-aed4-ef409f209cf5","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"DHA6OWMQ42PGF1T7","secret_key":"xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"test-zone1.rgw.log:notif"} 2021-02-11T09:46:14.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:14.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:14.802 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:46:14.884 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:14.884 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:22.982 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"f2716509-104d-423b-ba5c-db1bd74642ab","epoch":1,"predecessor_uuid":"8ab84e61-9000-48cf-8393-6e16e4da90b0","sync_status":[],"period_map":{"id":"f2716509-104d-423b-ba5c-db1bd74642ab","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","zones":[{"id":"a3c4910a-4529-4294-aed4-ef409f209cf5","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"a3c4910a-4529-4294-aed4-ef409f209cf5","val":2693415762}]},"master_zonegroup":"test-zonegroup","master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","realm_name":"test-realm","realm_epoch":2} 2021-02-11T09:46:22.983 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-02-11T09:46:22.983 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T09:46:23.025 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2021-02-11T09:46:23.024+0000 7fe48bfff700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 17122) UID: 0 2021-02-11T09:46:23.025 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2021-02-11T09:46:23.024+0000 7fe4afdde0c0 -1 shutting down 2021-02-11T09:46:27.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-02-11T09:46:29.084 INFO:tasks.rgw.c1.client.0:Stopped 2021-02-11T09:46:29.084 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-02-11T09:46:29.085 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-02-11T09:46:29.088 INFO:tasks.rgw.c1.client.0:Started 2021-02-11T09:46:29.088 DEBUG:teuthology.orchestra.run.smithi026:> curl http://smithi026.front.sepia.ceph.com:8000 2021-02-11T09:46:29.103 INFO:teuthology.orchestra.run.smithi026.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:46:29.104 INFO:teuthology.orchestra.run.smithi026.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:46:29.108 INFO:teuthology.orchestra.run.smithi026.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi026.front.sepia.ceph.com port 8000: Connection refused 2021-02-11T09:46:29.108 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T09:46:30.109 DEBUG:teuthology.orchestra.run.smithi026:> curl http://smithi026.front.sepia.ceph.com:8000 2021-02-11T09:46:30.123 INFO:teuthology.orchestra.run.smithi026.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:46:30.123 INFO:teuthology.orchestra.run.smithi026.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:46:30.129 INFO:teuthology.orchestra.run.smithi026.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2021-02-11T09:46:30.130 INFO:teuthology.orchestra.run.smithi026.stdout:anonymous 2021-02-11T09:46:30.130 INFO:tasks.rgw_multisite:creating admin user.. 2021-02-11T09:46:30.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'DHA6OWMQ42PGF1T7', '--secret', 'xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:30.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'DHA6OWMQ42PGF1T7', '--secret', 'xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:30.131 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key DHA6OWMQ42PGF1T7 --secret xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:46:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:30.414 INFO:teuthology.orchestra.run.smithi026.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"DHA6OWMQ42PGF1T7","secret_key":"xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":"true","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2021-02-11T09:46:30.448 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-02-11T09:46:30.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi026.front.sepia.ceph.com:8000', '--access-key', 'DHA6OWMQ42PGF1T7', '--secret', 'xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:30.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi026.front.sepia.ceph.com:8000', '--access-key', 'DHA6OWMQ42PGF1T7', '--secret', 'xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:30.448 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi026.front.sepia.ceph.com:8000 --access-key DHA6OWMQ42PGF1T7 --secret xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:46:30.490 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:30.491 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:30.621 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"52cf47b0-3540-4b08-a664-7ef37f763895","name":"test-realm","current_period":"f2716509-104d-423b-ba5c-db1bd74642ab","epoch":2} 2021-02-11T09:46:30.621 INFO:tasks.rgw_multisite:creating zone.. 2021-02-11T09:46:30.621 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'DHA6OWMQ42PGF1T7', '--secret', 'xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o', '--endpoints', 'http://smithi133.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:30.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'DHA6OWMQ42PGF1T7', '--secret', 'xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o', '--endpoints', 'http://smithi133.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:30.621 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key DHA6OWMQ42PGF1T7 --secret xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o --endpoints http://smithi133.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:46:30.705 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:30.705 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:30.771 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:30.770+0000 7efd68471080 0 failed reading obj info from .rgw.root:zone_info.a3c4910a-4529-4294-aed4-ef409f209cf5: (2) No such file or directory 2021-02-11T09:46:30.772 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:46:30.770+0000 7efd68471080 0 WARNING: could not read zone params for zone id=a3c4910a-4529-4294-aed4-ef409f209cf5 name=test-zone1 2021-02-11T09:46:30.784 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"DHA6OWMQ42PGF1T7","secret_key":"xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T09:46:30.784 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2021-02-11T09:46:31.546 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'test-zone2.rgw.buckets.data' created 2021-02-11T09:46:31.557 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2021-02-11T09:46:32.562 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2021-02-11T09:46:32.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:32.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:32.584 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:46:32.618 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:32.618 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:32.646 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"DHA6OWMQ42PGF1T7","secret_key":"xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T09:46:32.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:32.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:32.647 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:46:32.728 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:32.729 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:32.933 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-02-11T09:46:40.721 INFO:teuthology.orchestra.run.smithi133.stderr:Sending period to new master zone a3c4910a-4529-4294-aed4-ef409f209cf5 2021-02-11T09:46:40.823 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"f2716509-104d-423b-ba5c-db1bd74642ab","epoch":2,"predecessor_uuid":"8ab84e61-9000-48cf-8393-6e16e4da90b0","sync_status":[],"period_map":{"id":"f2716509-104d-423b-ba5c-db1bd74642ab","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","zones":[{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"a3c4910a-4529-4294-aed4-ef409f209cf5","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","val":3299024572},{"key":"a3c4910a-4529-4294-aed4-ef409f209cf5","val":2693415762}]},"master_zonegroup":"test-zonegroup","master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","realm_name":"test-realm","realm_epoch":2} 2021-02-11T09:46:40.824 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-02-11T09:46:40.824 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T09:46:40.870 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2021-02-11T09:46:40.870+0000 7f93767fc700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 17150) UID: 0 2021-02-11T09:46:40.870 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2021-02-11T09:46:40.870+0000 7f93929b40c0 -1 shutting down 2021-02-11T09:46:44.298 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2021-02-11T09:46:46.925 INFO:tasks.rgw.c2.client.0:Stopped 2021-02-11T09:46:46.926 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-02-11T09:46:46.926 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-02-11T09:46:46.929 INFO:tasks.rgw.c2.client.0:Started 2021-02-11T09:46:46.929 DEBUG:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8000 2021-02-11T09:46:46.944 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:46:46.944 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:46:46.949 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi133.front.sepia.ceph.com port 8000: Connection refused 2021-02-11T09:46:46.949 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T09:46:47.950 DEBUG:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8000 2021-02-11T09:46:47.964 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:46:47.964 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:46:47.970 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2021-02-11T09:46:47.971 INFO:teuthology.orchestra.run.smithi133.stdout:anonymous 2021-02-11T09:46:47.972 INFO:tasks.rgw_multisite:creating zone.. 2021-02-11T09:46:47.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'DHA6OWMQ42PGF1T7', '--secret', 'xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o', '--endpoints', 'http://smithi026.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:47.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'DHA6OWMQ42PGF1T7', '--secret', 'xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o', '--endpoints', 'http://smithi026.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:47.972 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key DHA6OWMQ42PGF1T7 --secret xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o --endpoints http://smithi026.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:46:48.017 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:48.017 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:48.081 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:48.080+0000 7fd4e9fdf080 0 failed reading obj info from .rgw.root:zone_info.2f5f3613-ec60-4b48-a757-91ddd4a3ec48: (2) No such file or directory 2021-02-11T09:46:48.081 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:46:48.080+0000 7fd4e9fdf080 0 WARNING: could not read zone params for zone id=2f5f3613-ec60-4b48-a757-91ddd4a3ec48 name=test-zone2 2021-02-11T09:46:48.094 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"30c2afa9-98cd-42de-acef-f4a612d65cea","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"DHA6OWMQ42PGF1T7","secret_key":"xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","notif_pool":"test-zone3.rgw.log:notif"} 2021-02-11T09:46:48.094 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2021-02-11T09:46:48.742 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'test-zone3.rgw.buckets.data' created 2021-02-11T09:46:48.764 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2021-02-11T09:46:49.752 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2021-02-11T09:46:49.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:49.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:49.784 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:46:49.817 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:49.817 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:49.848 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"30c2afa9-98cd-42de-acef-f4a612d65cea","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"DHA6OWMQ42PGF1T7","secret_key":"xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"test-zone3.rgw.log:notif"} 2021-02-11T09:46:49.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:49.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:46:49.849 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:46:49.932 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:49.932 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:50.003 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2021-02-11T09:46:58.026 INFO:teuthology.orchestra.run.smithi026.stderr:Sending period to new master zone a3c4910a-4529-4294-aed4-ef409f209cf5 2021-02-11T09:46:58.138 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"f2716509-104d-423b-ba5c-db1bd74642ab","epoch":3,"predecessor_uuid":"8ab84e61-9000-48cf-8393-6e16e4da90b0","sync_status":[],"period_map":{"id":"f2716509-104d-423b-ba5c-db1bd74642ab","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","zones":[{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"30c2afa9-98cd-42de-acef-f4a612d65cea","name":"test-zone3","endpoints":["http://smithi026.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"a3c4910a-4529-4294-aed4-ef409f209cf5","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","val":3299024572},{"key":"30c2afa9-98cd-42de-acef-f4a612d65cea","val":2754623789},{"key":"a3c4910a-4529-4294-aed4-ef409f209cf5","val":2693415762}]},"master_zonegroup":"test-zonegroup","master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","realm_name":"test-realm","realm_epoch":2} 2021-02-11T09:46:58.139 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-02-11T09:46:58.139 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T09:46:58.180 INFO:tasks.rgw.c1.client.1.smithi026.stdout:2021-02-11T09:46:58.180+0000 7f089ffff700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 17124) UID: 0 2021-02-11T09:46:58.181 INFO:tasks.rgw.c1.client.1.smithi026.stdout:2021-02-11T09:46:58.180+0000 7f08c3dfc0c0 -1 shutting down 2021-02-11T09:47:01.353 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-02-11T09:47:04.241 INFO:tasks.rgw.c1.client.1:Stopped 2021-02-11T09:47:04.241 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-02-11T09:47:04.242 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-02-11T09:47:04.244 INFO:tasks.rgw.c1.client.1:Started 2021-02-11T09:47:04.244 DEBUG:teuthology.orchestra.run.smithi026:> curl http://smithi026.front.sepia.ceph.com:8001 2021-02-11T09:47:04.262 INFO:teuthology.orchestra.run.smithi026.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:47:04.262 INFO:teuthology.orchestra.run.smithi026.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:47:04.266 INFO:teuthology.orchestra.run.smithi026.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi026.front.sepia.ceph.com port 8001: Connection refused 2021-02-11T09:47:04.266 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T09:47:05.268 DEBUG:teuthology.orchestra.run.smithi026:> curl http://smithi026.front.sepia.ceph.com:8001 2021-02-11T09:47:05.284 INFO:teuthology.orchestra.run.smithi026.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:47:05.284 INFO:teuthology.orchestra.run.smithi026.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:47:05.291 INFO:teuthology.orchestra.run.smithi026.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2021-02-11T09:47:05.292 INFO:teuthology.orchestra.run.smithi026.stdout:anonymous 2021-02-11T09:47:05.293 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-02-11T09:47:05.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi026.front.sepia.ceph.com:8000', '--access-key', 'DHA6OWMQ42PGF1T7', '--secret', 'xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o', '--url', 'http://smithi026.front.sepia.ceph.com:8000', '--access-key', 'DHA6OWMQ42PGF1T7', '--secret', 'xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:47:05.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi026.front.sepia.ceph.com:8000', '--access-key', 'DHA6OWMQ42PGF1T7', '--secret', 'xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o', '--url', 'http://smithi026.front.sepia.ceph.com:8000', '--access-key', 'DHA6OWMQ42PGF1T7', '--secret', 'xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:47:05.293 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi026.front.sepia.ceph.com:8000 --access-key DHA6OWMQ42PGF1T7 --secret xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o --url http://smithi026.front.sepia.ceph.com:8000 --access-key DHA6OWMQ42PGF1T7 --secret xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:47:05.344 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:05.345 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:05.392 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2021-02-11T09:47:05.390+0000 7f601d60a700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.52cf47b0-3540-4b08-a664-7ef37f763895.control err=-107 2021-02-11T09:47:05.416 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"52cf47b0-3540-4b08-a664-7ef37f763895","name":"test-realm","current_period":"f2716509-104d-423b-ba5c-db1bd74642ab","epoch":2} 2021-02-11T09:47:05.416 INFO:tasks.rgw_multisite:creating zone.. 2021-02-11T09:47:05.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'DHA6OWMQ42PGF1T7', '--secret', 'xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o', '--endpoints', 'http://smithi133.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:47:05.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'DHA6OWMQ42PGF1T7', '--secret', 'xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o', '--endpoints', 'http://smithi133.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:47:05.417 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key DHA6OWMQ42PGF1T7 --secret xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o --endpoints http://smithi133.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:47:05.466 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:05.466 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:05.535 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:47:05.534+0000 7f52f1d3e080 0 failed reading obj info from .rgw.root:zone_info.30c2afa9-98cd-42de-acef-f4a612d65cea: (2) No such file or directory 2021-02-11T09:47:05.535 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:47:05.534+0000 7f52f1d3e080 0 WARNING: could not read zone params for zone id=30c2afa9-98cd-42de-acef-f4a612d65cea name=test-zone3 2021-02-11T09:47:05.536 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:47:05.534+0000 7f52f1d3e080 0 failed reading obj info from .rgw.root:zone_info.a3c4910a-4529-4294-aed4-ef409f209cf5: (2) No such file or directory 2021-02-11T09:47:05.537 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:47:05.534+0000 7f52f1d3e080 0 WARNING: could not read zone params for zone id=a3c4910a-4529-4294-aed4-ef409f209cf5 name=test-zone1 2021-02-11T09:47:05.549 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"DHA6OWMQ42PGF1T7","secret_key":"xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","notif_pool":"test-zone4.rgw.log:notif"} 2021-02-11T09:47:05.549 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2021-02-11T09:47:06.527 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'test-zone4.rgw.buckets.data' created 2021-02-11T09:47:06.546 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2021-02-11T09:47:07.060 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-02-11T09:47:07.539 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2021-02-11T09:47:07.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:47:07.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:47:07.573 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:47:07.608 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:07.609 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:07.634 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"DHA6OWMQ42PGF1T7","secret_key":"xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"","notif_pool":"test-zone4.rgw.log:notif"} 2021-02-11T09:47:07.634 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:47:07.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:47:07.635 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:47:07.713 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:07.713 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:15.747 INFO:teuthology.orchestra.run.smithi133.stderr:Sending period to new master zone a3c4910a-4529-4294-aed4-ef409f209cf5 2021-02-11T09:47:15.850 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"f2716509-104d-423b-ba5c-db1bd74642ab","epoch":4,"predecessor_uuid":"8ab84e61-9000-48cf-8393-6e16e4da90b0","sync_status":[],"period_map":{"id":"f2716509-104d-423b-ba5c-db1bd74642ab","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","zones":[{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"30c2afa9-98cd-42de-acef-f4a612d65cea","name":"test-zone3","endpoints":["http://smithi026.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"a3c4910a-4529-4294-aed4-ef409f209cf5","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","val":3299024572},{"key":"30c2afa9-98cd-42de-acef-f4a612d65cea","val":2754623789},{"key":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","val":3684997273},{"key":"a3c4910a-4529-4294-aed4-ef409f209cf5","val":2693415762}]},"master_zonegroup":"test-zonegroup","master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","realm_name":"test-realm","realm_epoch":2} 2021-02-11T09:47:15.851 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-02-11T09:47:15.851 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T09:47:15.893 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2021-02-11T09:47:15.894+0000 7f9948c39700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 17152) UID: 0 2021-02-11T09:47:15.893 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2021-02-11T09:47:15.894+0000 7f99607940c0 -1 shutting down 2021-02-11T09:47:18.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2021-02-11T09:47:21.953 INFO:tasks.rgw.c2.client.1:Stopped 2021-02-11T09:47:24.442 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-02-11T09:47:24.442 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-02-11T09:47:24.445 INFO:tasks.rgw.c2.client.1:Started 2021-02-11T09:47:24.445 DEBUG:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8001 2021-02-11T09:47:24.462 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:47:24.463 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:47:24.467 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi133.front.sepia.ceph.com port 8001: Connection refused 2021-02-11T09:47:24.468 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T09:47:25.148 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2021-02-11T09:47:25.469 DEBUG:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8001 2021-02-11T09:47:25.482 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:47:25.482 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:47:25.486 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 33500 0 --:--:-- --:--:-- --:--:-- 33500 2021-02-11T09:47:25.487 INFO:teuthology.orchestra.run.smithi133.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-006024fd2d-10e6-test-zone4","HostId":"10e6-test-zone4-test-zonegroup"} 2021-02-11T09:47:25.487 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2021-02-11T09:47:25.487 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2021-02-11T09:47:25.544 INFO:tasks.rgw_multisite_tests:creating test user.. 2021-02-11T09:47:25.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:47:25.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:47:25.545 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:47:25.590 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:25.590 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:25.808 INFO:teuthology.orchestra.run.smithi026.stdout:{"user_id":"rgw-multisite-test-user","display_name":"Multisite Test User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"SCC8BVV0TN0KCVAE0QNZ","secret_key":"JZ5aTMLuHUXVVbvzZ8ikAswZtxdNi4C4VweAyXlu"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2021-02-11T09:47:25.837 INFO:rgw_multi.tests:meta checkpoint 2021-02-11T09:47:25.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:25.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:25.837 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:25.917 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:25.917 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:26.110 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f2716509-104d-423b-ba5c-db1bd74642ab 2021-02-11T09:47:26.152 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613036845.807024_48.1","last_update":"2021-02-11T09:47:25.807024Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:47:26.152 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:47:26.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:26.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:26.153 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:26.194 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:26.194 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:26.418 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:26.419 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:47:31.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:31.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:31.421 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:31.459 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:31.459 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:31.697 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:31.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:47:36.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:36.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:36.699 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:36.740 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:36.740 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:36.968 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:36.968 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:47:41.968 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:41.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:41.969 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:42.011 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:42.011 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:42.250 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:42.250 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:47:47.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:47.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:47.252 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:47.292 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:47.292 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:47.529 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:47.529 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:47:52.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:52.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:52.530 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:52.574 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:52.575 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:52.815 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:52.815 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:47:57.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:57.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:57.817 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:57.860 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:57.860 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:58.078 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:58.078 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:48:03.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:03.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:03.080 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:03.123 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:03.123 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:03.346 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:48:03.347 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:48:08.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:08.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:08.348 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:08.389 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:08.389 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:08.614 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:48:08.614 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:48:13.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:13.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:13.616 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:13.660 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:13.660 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:13.888 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:48:13.888 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:48:18.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:18.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:18.889 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:18.931 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:18.931 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:19.146 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:48:19.146 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:48:24.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:24.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:24.148 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:24.189 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:24.189 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:24.414 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:48:24.415 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:48:29.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:29.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:29.417 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:29.458 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:29.458 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:29.691 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:48:29.692 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:48:34.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:34.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:34.693 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:34.734 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:34.735 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:34.959 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:48:34.959 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:48:39.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:39.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:39.961 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:40.003 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:40.003 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:40.223 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:48:40.223 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:48:45.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:45.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:45.225 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:45.265 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:45.265 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:45.500 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:48:45.501 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:48:50.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:50.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:50.502 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:50.541 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:50.541 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:50.758 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:48:50.759 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:48:55.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:55.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:55.760 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:55.800 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:55.800 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:56.022 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:48:56.023 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:49:01.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:01.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:01.025 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:01.065 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:01.066 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:01.313 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:01.313 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:49:06.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:06.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:06.315 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:06.357 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:06.357 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:06.591 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:06.591 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:49:11.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:14.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:14.333 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:14.373 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:14.373 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:14.601 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:14.602 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036845.807024_48.1 target= 2021-02-11T09:49:19.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:19.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:19.603 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:19.641 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:19.641 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:19.858 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036845.807024_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:47:25.807024Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:19.858 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:49:19.858 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:49:19.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:19.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:19.859 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:19.900 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:19.901 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:20.135 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036845.807024_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:47:25.807024Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:20.135 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:49:20.135 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:49:20.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:20.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:20.136 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:20.176 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:20.176 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:20.408 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036845.807024_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:47:25.807024Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:20.408 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:49:20.408 INFO:rgw_multi.tests:meta checkpoint 2021-02-11T09:49:20.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:20.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:20.409 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:20.447 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:20.447 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:20.642 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f2716509-104d-423b-ba5c-db1bd74642ab 2021-02-11T09:49:20.679 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613036845.807024_48.1","last_update":"2021-02-11T09:47:25.807024Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:49:20.680 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:49:20.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:20.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:20.680 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:20.720 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:20.720 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:20.939 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036845.807024_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:47:25.807024Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:20.939 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:49:20.939 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:49:20.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:20.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:20.940 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:20.981 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:20.981 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:21.234 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036845.807024_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:47:25.807024Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:21.234 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:49:21.234 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:49:21.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:21.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:21.235 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:21.276 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:21.276 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:21.498 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036845.807024_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:47:25.807024Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:21.499 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:49:21.499 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2021-02-11T09:49:21.508 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-1 2021-02-11T09:49:24.805 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-2 2021-02-11T09:49:28.622 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-3 2021-02-11T09:49:28.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:28.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:28.693 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:28.733 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:28.734 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:28.931 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f2716509-104d-423b-ba5c-db1bd74642ab 2021-02-11T09:49:28.974 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613036968.672641_67.1","last_update":"2021-02-11T09:49:28.672641Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:49:28.974 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:49:28.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:28.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:28.975 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:29.014 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:29.015 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:29.245 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036968.672641_67.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:28.672641Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:29.246 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:49:29.246 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:49:29.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:29.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:29.246 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:29.287 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:29.287 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:29.535 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036968.672641_67.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:28.672641Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:29.536 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:49:29.536 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:49:29.536 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:29.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:29.536 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:29.577 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:29.578 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:29.796 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036968.672641_67.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:28.672641Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:29.796 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:49:31.777 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2021-02-11T09:49:31.777 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-4 2021-02-11T09:49:32.571 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-5 2021-02-11T09:49:32.923 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-6 2021-02-11T09:49:33.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:33.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:33.604 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:33.641 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:33.641 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:33.831 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f2716509-104d-423b-ba5c-db1bd74642ab 2021-02-11T09:49:33.868 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613036973.589655_80.1","last_update":"2021-02-11T09:49:33.589655Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:49:33.868 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:49:33.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:33.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:33.868 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:33.909 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:33.909 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:34.155 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036973.589655_80.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:33.589655Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:34.155 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:49:34.156 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:49:34.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:34.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:34.156 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:34.195 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:34.196 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:34.425 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036973.589655_80.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:33.589655Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:34.425 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:49:34.425 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:49:34.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:34.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:34.426 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:34.468 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:34.468 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:34.680 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036973.589655_80.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:33.589655Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:34.680 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:49:38.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:38.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:38.946 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:38.979 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:38.980 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:39.170 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f2716509-104d-423b-ba5c-db1bd74642ab 2021-02-11T09:49:39.208 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613036973.589655_80.1","last_update":"2021-02-11T09:49:33.589655Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:49:39.208 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:49:39.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:39.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:39.209 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:39.255 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:39.256 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:39.492 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036973.589655_80.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:33.589655Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:39.494 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:49:39.494 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:49:39.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:39.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:39.494 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:39.537 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:39.537 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:39.779 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036973.589655_80.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:33.589655Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:39.780 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:49:39.780 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:49:39.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:39.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:39.780 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:39.820 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:39.820 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:40.049 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036973.589655_80.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:33.589655Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:40.049 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:49:41.614 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2021-02-11T09:49:41.614 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-7 2021-02-11T09:49:41.645 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-8 2021-02-11T09:49:41.715 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-9 2021-02-11T09:49:41.774 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:41.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:41.774 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:42.002 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f2716509-104d-423b-ba5c-db1bd74642ab 2021-02-11T09:49:42.040 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613036981.757312_94.1","last_update":"2021-02-11T09:49:41.757312Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:49:42.040 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:49:42.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:42.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:42.041 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:42.082 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:42.082 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:42.324 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036981.757312_94.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:41.757312Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:42.325 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:49:42.325 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:49:42.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:42.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:42.325 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:42.366 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:42.367 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:42.615 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036981.757312_94.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:41.757312Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:42.615 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:49:42.615 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:49:42.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:42.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:42.616 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:42.650 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:42.650 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:42.875 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036981.757312_94.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:41.757312Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:42.875 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:49:45.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:56.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:56.496 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:56.535 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:56.535 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:56.733 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f2716509-104d-423b-ba5c-db1bd74642ab 2021-02-11T09:49:56.767 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613036985.070217_101.1","last_update":"2021-02-11T09:49:45.070217Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:49:56.767 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:49:56.767 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:56.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:56.768 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:56.809 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:56.810 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:57.042 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:57.043 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:49:57.043 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:49:57.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:57.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:57.043 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:57.087 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:57.087 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:57.333 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:57.334 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:49:57.334 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:49:57.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:57.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:57.334 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:57.373 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:57.374 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:57.596 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:57.596 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:49:59.213 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2021-02-11T09:49:59.213 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-10 2021-02-11T09:49:59.856 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-11 2021-02-11T09:50:00.028 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-12 2021-02-11T09:50:00.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:00.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:00.675 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:00.732 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:00.732 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:00.986 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f2716509-104d-423b-ba5c-db1bd74642ab 2021-02-11T09:50:01.023 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613036985.070217_101.1","last_update":"2021-02-11T09:49:45.070217Z"},{"marker":"1_1613037000.074230_63.1","last_update":"2021-02-11T09:50:00.074230Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:50:01.024 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:50:01.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:01.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:01.024 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:01.069 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:01.069 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:01.273 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037000.074230_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:00.074230Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:01.274 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:50:01.274 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:50:01.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:01.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:01.274 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:01.316 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:01.316 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:01.562 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037000.074230_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:00.074230Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:01.562 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:50:01.563 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:50:01.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:01.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:01.563 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:01.603 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:01.603 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:01.824 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037000.074230_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:00.074230Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:01.825 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:50:01.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:01.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:01.825 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:01.913 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:01.913 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:02.129 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:02.130+0000 7f33fdca7080 1 bucket sync caught up with source: 2021-02-11T09:50:02.129 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.7.5, 00000000001.8.5, , ] 2021-02-11T09:50:02.129 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.7.5, 00000000001.8.5, , ] 2021-02-11T09:50:02.129 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:02.130+0000 7f33fdca7080 0 bucket checkpoint complete 2021-02-11T09:50:02.162 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-10 zones={test-zone2, test-zone1} 2021-02-11T09:50:03.012 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:03.021 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:50:03.021 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:50:03.021 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:50:03.021 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:03.021 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:03.028 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:50:03.028 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:50:03.028 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:50:03.028 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:03.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:50:03.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:50:03.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:50:03.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:50:03.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:50:03.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:50:03.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:50:03.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:50:03.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:50:03.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:03.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:04.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:04.122 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-10 zones={test-zone2, test-zone1} 2021-02-11T09:50:04.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:04.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:04.123 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:04.167 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:04.167 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:04.390 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:04.388+0000 7f76f2b71080 1 bucket sync caught up with source: 2021-02-11T09:50:04.390 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.7.5, 00000000001.8.5, , ] 2021-02-11T09:50:04.390 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.7.5, 00000000001.8.5, , ] 2021-02-11T09:50:04.391 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:04.388+0000 7f76f2b71080 0 bucket checkpoint complete 2021-02-11T09:50:04.415 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-10 zones={test-zone3, test-zone1} 2021-02-11T09:50:04.820 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:04.825 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:50:04.825 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:50:04.826 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:50:04.826 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:04.826 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:04.830 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:50:04.830 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:50:04.830 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:50:04.831 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:04.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:50:04.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:50:04.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:50:05.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:50:05.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:50:05.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:50:05.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:50:05.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:50:05.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:50:05.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:05.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:05.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:05.901 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-10 zones={test-zone3, test-zone1} 2021-02-11T09:50:05.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:05.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:05.902 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:05.949 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:05.950 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:06.159 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:06.162+0000 7f538912e080 1 bucket sync caught up with source: 2021-02-11T09:50:06.159 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.7.5, 00000000001.8.5, , ] 2021-02-11T09:50:06.159 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.7.5, 00000000001.8.5, , ] 2021-02-11T09:50:06.159 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:06.162+0000 7f538912e080 0 bucket checkpoint complete 2021-02-11T09:50:06.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:06.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:06.190 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:06.240 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:06.240 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:06.472 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:06.472+0000 7f2084da8080 1 bucket sync caught up with source: 2021-02-11T09:50:06.472 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.3.5, , 00000000001.14.5, , , , 00000000001.5.5, 00000000001.10.5, , ] 2021-02-11T09:50:06.472 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.3.5, , 00000000001.14.5, , , , 00000000001.5.5, 00000000001.10.5, , ] 2021-02-11T09:50:06.472 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:06.472+0000 7f2084da8080 0 bucket checkpoint complete 2021-02-11T09:50:06.504 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-11 zones={test-zone1, test-zone2} 2021-02-11T09:50:06.713 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:06.720 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:50:06.720 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:50:06.720 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:50:06.720 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:06.721 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:06.726 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:50:06.726 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:50:06.726 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:50:06.726 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:06.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:50:06.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:50:06.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:50:07.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:50:07.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:50:07.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:50:07.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:50:07.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:50:07.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:50:07.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:07.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:07.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:07.809 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-11 zones={test-zone1, test-zone2} 2021-02-11T09:50:07.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:07.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:07.810 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:07.855 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:07.856 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:08.071 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:08.072+0000 7f7bdabee080 1 bucket sync caught up with source: 2021-02-11T09:50:08.071 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.3.5, , 00000000001.14.5, , , , 00000000001.5.5, 00000000001.10.5, , ] 2021-02-11T09:50:08.071 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.3.5, , 00000000001.14.5, , , , 00000000001.5.5, 00000000001.10.5, , ] 2021-02-11T09:50:08.071 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:08.072+0000 7f7bdabee080 0 bucket checkpoint complete 2021-02-11T09:50:08.107 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-11 zones={test-zone3, test-zone2} 2021-02-11T09:50:08.559 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:08.565 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:50:08.565 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:50:08.565 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:50:08.565 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:08.566 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:08.569 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:50:08.569 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:50:08.569 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:50:08.569 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:08.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:50:08.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:50:08.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:50:08.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:50:08.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:50:09.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:50:09.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:50:09.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:50:09.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:50:09.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:09.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:09.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:09.648 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-11 zones={test-zone3, test-zone2} 2021-02-11T09:50:09.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:09.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:09.649 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:09.920 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:09.920+0000 7f20ddc85080 1 bucket sync caught up with source: 2021-02-11T09:50:09.920 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.5.5, , 00000000001.9.5, , , , 00000000001.6.5, 00000000001.10.5, , ] 2021-02-11T09:50:09.920 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.5.5, , 00000000001.9.5, , , , 00000000001.6.5, 00000000001.10.5, , ] 2021-02-11T09:50:09.920 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:09.920+0000 7f20ddc85080 0 bucket checkpoint complete 2021-02-11T09:50:09.951 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-12 zones={test-zone1, test-zone3} 2021-02-11T09:50:10.102 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:10.106 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:50:10.106 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:50:10.107 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:50:10.107 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:10.107 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:10.110 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:50:10.111 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:50:10.111 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:50:10.111 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:10.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:50:10.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:50:10.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:50:10.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:50:10.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:50:10.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:50:10.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:50:10.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:50:10.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:50:10.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:10.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:11.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:11.197 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-12 zones={test-zone1, test-zone3} 2021-02-11T09:50:11.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:11.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:11.198 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:11.245 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:11.245 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:11.467 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:11.470+0000 7f2963c7c080 1 bucket sync caught up with source: 2021-02-11T09:50:11.467 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000001.5.5, , 00000000001.9.5, , , , 00000000001.6.5, 00000000001.10.5, , ] 2021-02-11T09:50:11.467 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000001.5.5, , 00000000001.9.5, , , , 00000000001.6.5, 00000000001.10.5, , ] 2021-02-11T09:50:11.467 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:11.470+0000 7f2963c7c080 0 bucket checkpoint complete 2021-02-11T09:50:11.500 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-12 zones={test-zone2, test-zone3} 2021-02-11T09:50:11.921 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:11.927 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:50:11.927 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:50:11.927 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:50:11.927 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:11.927 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:11.933 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:50:11.933 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:50:11.933 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:50:11.934 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:11.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:50:11.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:50:12.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:50:12.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:50:12.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:50:12.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:50:12.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:50:12.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:50:12.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:50:12.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:12.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:12.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:13.017 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-12 zones={test-zone2, test-zone3} 2021-02-11T09:50:13.018 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2021-02-11T09:50:13.018 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-13 2021-02-11T09:50:13.715 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-14 2021-02-11T09:50:13.773 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-15 2021-02-11T09:50:13.848 INFO:tasks.rgw.c1.client.1.smithi026.stdout:2021-02-11T09:50:13.848+0000 7f68e77f6700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:50:13.849 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2021-02-11T09:50:13.850+0000 7f62237fe700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:50:13.849 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2021-02-11T09:50:13.850+0000 7f8550fe9700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:50:13.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:13.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:13.971 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:14.012 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:14.012 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:14.203 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f2716509-104d-423b-ba5c-db1bd74642ab 2021-02-11T09:50:14.239 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613036985.070217_101.1","last_update":"2021-02-11T09:49:45.070217Z"},{"marker":"1_1613037013.812144_178.1","last_update":"2021-02-11T09:50:13.812144Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:50:14.239 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:50:14.239 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:14.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:14.240 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:14.280 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:14.280 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:14.476 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037013.804452_176.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:13.804452Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:14.476 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037013.812144_178.1 target=1_1613037013.804452_176.1 2021-02-11T09:50:19.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:20.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:20.872 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:20.912 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:20.912 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:21.148 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037013.812144_178.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:13.812144Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:21.149 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:50:21.149 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:50:21.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:21.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:21.149 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:21.190 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:21.190 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:21.428 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037013.812144_178.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:13.812144Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:21.428 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:50:21.428 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:50:21.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:21.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:21.429 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:21.470 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:21.470 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:21.672 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037013.812144_178.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:13.812144Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:21.673 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:50:21.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:21.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:21.673 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:21.758 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:21.758 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:21.972 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:21.974+0000 7f9e42dfd080 1 bucket sync caught up with source: 2021-02-11T09:50:21.973 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2021-02-11T09:50:21.973 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000001.6.5, , , , , , , , , ] 2021-02-11T09:50:21.973 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:21.974+0000 7f9e42dfd080 0 bucket checkpoint complete 2021-02-11T09:50:22.010 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-13 zones={test-zone2, test-zone1} 2021-02-11T09:50:22.402 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:22.409 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:22.409 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:22.459 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:22.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:22.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:22.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:22.778 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-13 zones={test-zone2, test-zone1} 2021-02-11T09:50:22.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:22.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:22.779 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:22.825 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:22.825 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:23.036 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:23.036+0000 7fcab55ea080 1 bucket sync caught up with source: 2021-02-11T09:50:23.036 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2021-02-11T09:50:23.036 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.6.5, , , , , , , , , ] 2021-02-11T09:50:23.036 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:23.036+0000 7fcab55ea080 0 bucket checkpoint complete 2021-02-11T09:50:23.066 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-13 zones={test-zone3, test-zone1} 2021-02-11T09:50:24.926 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:24.933 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:24.933 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:24.982 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:25.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:25.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:25.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:25.306 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-13 zones={test-zone3, test-zone1} 2021-02-11T09:50:25.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:25.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:25.306 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:25.354 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:25.354 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:25.581 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:25.586+0000 7fc4ca673080 1 bucket sync caught up with source: 2021-02-11T09:50:25.582 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2021-02-11T09:50:25.582 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000001.6.5, , , , , , , , , ] 2021-02-11T09:50:25.582 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:25.586+0000 7fc4ca673080 0 bucket checkpoint complete 2021-02-11T09:50:25.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:25.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:25.617 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:25.666 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:25.667 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:25.876 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:25.876+0000 7f730172c080 1 bucket sync caught up with source: 2021-02-11T09:50:25.876 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.14.5, , , , , , , , , ] 2021-02-11T09:50:25.877 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.14.5, , , , , , , , , ] 2021-02-11T09:50:25.877 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:25.876+0000 7f730172c080 0 bucket checkpoint complete 2021-02-11T09:50:25.917 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-14 zones={test-zone1, test-zone2} 2021-02-11T09:50:26.864 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:26.872 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:26.872 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:26.918 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:26.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:27.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:27.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:27.245 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-14 zones={test-zone1, test-zone2} 2021-02-11T09:50:27.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:27.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:27.246 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:27.296 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:27.296 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:27.502 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:27.504+0000 7fab5d35a080 1 bucket sync caught up with source: 2021-02-11T09:50:27.503 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.14.5, , , , , , , , , ] 2021-02-11T09:50:27.503 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.14.5, , , , , , , , , ] 2021-02-11T09:50:27.503 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:27.504+0000 7fab5d35a080 0 bucket checkpoint complete 2021-02-11T09:50:27.534 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-14 zones={test-zone3, test-zone2} 2021-02-11T09:50:28.410 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:28.416 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:28.417 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:28.466 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:28.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:28.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:28.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:28.789 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-14 zones={test-zone3, test-zone2} 2021-02-11T09:50:28.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:28.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:28.790 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:28.837 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:28.837 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:29.059 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:29.060+0000 7fe86ccbe080 1 bucket sync caught up with source: 2021-02-11T09:50:29.059 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2021-02-11T09:50:29.059 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.8.5, , , , , , , , , ] 2021-02-11T09:50:29.059 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:29.060+0000 7fe86ccbe080 0 bucket checkpoint complete 2021-02-11T09:50:29.088 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-15 zones={test-zone1, test-zone3} 2021-02-11T09:50:29.714 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:29.766 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:29.766 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:29.772 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:29.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:29.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:29.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:30.105 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-15 zones={test-zone1, test-zone3} 2021-02-11T09:50:30.106 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:30.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:30.106 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:30.154 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:30.154 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:30.367 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:30.370+0000 7f6dcfb6a080 1 bucket sync caught up with source: 2021-02-11T09:50:30.367 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2021-02-11T09:50:30.367 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000001.8.5, , , , , , , , , ] 2021-02-11T09:50:30.367 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:30.370+0000 7f6dcfb6a080 0 bucket checkpoint complete 2021-02-11T09:50:30.399 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-15 zones={test-zone2, test-zone3} 2021-02-11T09:50:30.835 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:30.883 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:30.883 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:30.890 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:50:30.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:30.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:31.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:50:31.257 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-15 zones={test-zone2, test-zone3} 2021-02-11T09:50:32.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:32.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:32.194 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:32.242 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:32.243 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:32.448 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:32.450+0000 7fb42edea080 1 bucket sync caught up with source: 2021-02-11T09:50:32.448 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2021-02-11T09:50:32.448 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2021-02-11T09:50:32.448 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:32.450+0000 7fb42edea080 0 bucket checkpoint complete 2021-02-11T09:50:32.481 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-13 zones={test-zone2, test-zone1} 2021-02-11T09:50:33.081 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:33.130 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:33.189 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-13 zones={test-zone2, test-zone1} 2021-02-11T09:50:33.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:33.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:33.190 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:33.236 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:33.236 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:33.460 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:33.460+0000 7f4d172c3080 1 bucket sync caught up with source: 2021-02-11T09:50:33.460 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2021-02-11T09:50:33.460 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2021-02-11T09:50:33.460 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:33.460+0000 7f4d172c3080 0 bucket checkpoint complete 2021-02-11T09:50:33.489 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-13 zones={test-zone3, test-zone1} 2021-02-11T09:50:35.076 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:35.084 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:35.182 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-13 zones={test-zone3, test-zone1} 2021-02-11T09:50:35.183 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:35.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:35.183 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:35.230 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:35.230 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:35.448 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:35.450+0000 7f52c8d21080 1 bucket sync caught up with source: 2021-02-11T09:50:35.448 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2021-02-11T09:50:35.449 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2021-02-11T09:50:35.449 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:35.450+0000 7f52c8d21080 0 bucket checkpoint complete 2021-02-11T09:50:35.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:35.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:36.000 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:36.050 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:36.050 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:36.264 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:36.264+0000 7fd274a9d080 1 bucket sync caught up with source: 2021-02-11T09:50:36.264 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.18.5, , , , , , , , , ] 2021-02-11T09:50:36.264 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2021-02-11T09:50:36.264 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:36.264+0000 7fd274a9d080 0 bucket checkpoint complete 2021-02-11T09:50:36.295 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-14 zones={test-zone1, test-zone2} 2021-02-11T09:50:38.101 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:38.108 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:38.206 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-14 zones={test-zone1, test-zone2} 2021-02-11T09:50:38.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:38.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:38.207 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:38.255 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:38.256 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:38.482 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:38.484+0000 7fc1a8168080 1 bucket sync caught up with source: 2021-02-11T09:50:38.482 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.18.5, , , , , , , , , ] 2021-02-11T09:50:38.483 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2021-02-11T09:50:38.483 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:38.484+0000 7fc1a8168080 0 bucket checkpoint complete 2021-02-11T09:50:38.517 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-14 zones={test-zone3, test-zone2} 2021-02-11T09:50:38.789 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:38.838 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:38.900 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-14 zones={test-zone3, test-zone2} 2021-02-11T09:50:38.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:38.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:38.917 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:38.960 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:38.960 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:39.158 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:39.160+0000 7f7acfa5d080 1 bucket sync caught up with source: 2021-02-11T09:50:39.158 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000002.10.5, , , , , , , , , ] 2021-02-11T09:50:39.158 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2021-02-11T09:50:39.158 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:39.160+0000 7f7acfa5d080 0 bucket checkpoint complete 2021-02-11T09:50:39.189 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-15 zones={test-zone1, test-zone3} 2021-02-11T09:50:39.778 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:39.822 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:39.882 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-15 zones={test-zone1, test-zone3} 2021-02-11T09:50:39.882 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:39.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:39.882 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:39.929 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:39.929 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:40.142 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:40.146+0000 7f02d66b0080 1 bucket sync caught up with source: 2021-02-11T09:50:40.142 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000002.10.5, , , , , , , , , ] 2021-02-11T09:50:40.142 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2021-02-11T09:50:40.142 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:40.146+0000 7f02d66b0080 0 bucket checkpoint complete 2021-02-11T09:50:40.177 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-15 zones={test-zone2, test-zone3} 2021-02-11T09:50:40.875 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:40.927 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:40.994 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-15 zones={test-zone2, test-zone3} 2021-02-11T09:50:40.994 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2021-02-11T09:50:40.995 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-16 2021-02-11T09:50:41.988 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-17 2021-02-11T09:50:42.722 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-18 2021-02-11T09:50:45.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:45.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:45.654 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:45.693 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:45.693 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:45.887 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f2716509-104d-423b-ba5c-db1bd74642ab 2021-02-11T09:50:45.924 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613036985.070217_101.1","last_update":"2021-02-11T09:49:45.070217Z"},{"marker":"1_1613037045.651083_200.1","last_update":"2021-02-11T09:50:45.651083Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:50:45.925 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:50:45.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:45.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:45.925 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:45.967 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:45.967 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:46.204 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037045.651083_200.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:45.651083Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:46.204 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:50:46.204 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:50:46.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:46.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:46.205 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:46.246 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:46.247 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:46.489 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037045.651083_200.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:45.651083Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:46.490 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:50:46.490 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:50:46.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:46.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:46.490 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:46.530 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:46.531 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:46.760 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037045.651083_200.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:45.651083Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:46.761 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:50:47.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:47.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:47.027 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:47.077 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:47.077 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:47.284 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:47.286+0000 7f6a8a8f6080 1 bucket sync caught up with source: 2021-02-11T09:50:47.285 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , 00000000002.12.10, , , , , ] 2021-02-11T09:50:47.285 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , 00000000002.12.10, , , , , ] 2021-02-11T09:50:47.285 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:47.286+0000 7f6a8a8f6080 0 bucket checkpoint complete 2021-02-11T09:50:47.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:47.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:47.315 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:47.367 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:47.367 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:47.582 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:47.583+0000 7f9cff525080 1 waiting for incremental sync to catch up: 2021-02-11T09:50:47.583 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000004.16.9, , , , , ] 2021-02-11T09:50:47.583 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000006.20.9, , , , , ] 2021-02-11T09:50:52.587 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:52.590+0000 7f9cff525080 1 bucket sync caught up with source: 2021-02-11T09:50:52.587 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000006.20.9, , , , , ] 2021-02-11T09:50:52.588 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000006.20.9, , , , , ] 2021-02-11T09:50:52.588 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:52.590+0000 7f9cff525080 0 bucket checkpoint complete 2021-02-11T09:50:52.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:52.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:52.622 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:52.670 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:52.670 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:52.886 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:52.889+0000 7efd638c3080 1 bucket sync caught up with source: 2021-02-11T09:50:52.887 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , 00000000006.20.9, , , , , ] 2021-02-11T09:50:52.887 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , 00000000006.20.9, , , , , ] 2021-02-11T09:50:52.887 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:50:52.889+0000 7efd638c3080 0 bucket checkpoint complete 2021-02-11T09:50:53.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:53.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:53.135 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:53.184 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:53.185 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:53.417 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:53.418+0000 7fa557ae9080 1 waiting for incremental sync to catch up: 2021-02-11T09:50:53.417 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , , , ] 2021-02-11T09:50:53.418 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000002.9.10, , , , , ] 2021-02-11T09:50:58.423 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:58.425+0000 7fa557ae9080 1 bucket sync caught up with source: 2021-02-11T09:50:58.423 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000006.17.9, , , , , ] 2021-02-11T09:50:58.423 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000002.9.10, , , , , ] 2021-02-11T09:50:58.423 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:58.425+0000 7fa557ae9080 0 bucket checkpoint complete 2021-02-11T09:50:58.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:58.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:58.459 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:58.506 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:58.507 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:58.728 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:58.729+0000 7f05f9f57080 1 bucket sync caught up with source: 2021-02-11T09:50:58.729 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000006.17.9, , , , , ] 2021-02-11T09:50:58.729 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000006.17.9, , , , , ] 2021-02-11T09:50:58.729 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:58.729+0000 7f05f9f57080 0 bucket checkpoint complete 2021-02-11T09:50:59.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:59.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:59.407 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:59.454 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:59.454 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:59.701 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:50:59.701+0000 7f25f256a080 1 waiting for incremental sync to catch up: 2021-02-11T09:50:59.702 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , , , ] 2021-02-11T09:50:59.702 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000002.26.10, , , , , ] 2021-02-11T09:51:04.706 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:04.704+0000 7f25f256a080 1 bucket sync caught up with source: 2021-02-11T09:51:04.706 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000006.34.9, , , , , ] 2021-02-11T09:51:04.706 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000002.26.10, , , , , ] 2021-02-11T09:51:04.706 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:04.704+0000 7f25f256a080 0 bucket checkpoint complete 2021-02-11T09:51:04.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:04.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:04.739 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:04.786 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:04.787 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:04.996 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:04.995+0000 7fc472223080 1 bucket sync caught up with source: 2021-02-11T09:51:04.996 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , 00000000006.34.9, , , , , ] 2021-02-11T09:51:04.997 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , 00000000006.34.9, , , , , ] 2021-02-11T09:51:04.997 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:04.995+0000 7fc472223080 0 bucket checkpoint complete 2021-02-11T09:51:08.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:08.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:08.234 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:08.287 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:08.287 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:08.498 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:08.495+0000 7fa8d931a080 1 bucket sync caught up with source: 2021-02-11T09:51:08.498 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000010.34.5, , 00000000003.10.9, , 00000000009.26.9, , , , , ] 2021-02-11T09:51:08.498 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000010.34.5, , 00000000003.10.9, , 00000000009.26.9, , , , , ] 2021-02-11T09:51:08.498 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:08.495+0000 7fa8d931a080 0 bucket checkpoint complete 2021-02-11T09:51:08.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:08.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:08.533 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:08.580 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:08.580 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:08.780+0000 7fd6765c2080 1 bucket sync caught up with source: 2021-02-11T09:51:08.780 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000010.34.5, , 00000000003.10.9, , 00000000009.26.9, , , , , ] 2021-02-11T09:51:08.781 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000010.34.5, , 00000000003.10.9, , 00000000009.26.9, , , , , ] 2021-02-11T09:51:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:08.780+0000 7fd6765c2080 0 bucket checkpoint complete 2021-02-11T09:51:08.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:08.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:08.806 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:08.854 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:08.854 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:09.083 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:09.083+0000 7f0c48cab080 1 bucket sync caught up with source: 2021-02-11T09:51:09.084 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000010.34.5, , 00000000003.10.9, , 00000000009.26.9, , , , , ] 2021-02-11T09:51:09.084 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000010.34.5, , 00000000003.10.9, , 00000000009.26.9, , , , , ] 2021-02-11T09:51:09.084 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:09.083+0000 7f0c48cab080 0 bucket checkpoint complete 2021-02-11T09:51:09.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:09.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:09.115 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:09.164 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:09.164 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:09.393 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:09.392+0000 7effe41c4080 1 bucket sync caught up with source: 2021-02-11T09:51:09.394 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000003.25.9, , 00000000010.26.5, , 00000000007.27.9, , , , , ] 2021-02-11T09:51:09.394 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000003.25.9, , 00000000010.26.5, , 00000000007.27.9, , , , , ] 2021-02-11T09:51:09.394 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:09.392+0000 7effe41c4080 0 bucket checkpoint complete 2021-02-11T09:51:09.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:09.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:09.433 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:09.481 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:09.481 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:09.700+0000 7fc93df32080 1 bucket sync caught up with source: 2021-02-11T09:51:09.702 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000003.25.9, , 00000000010.26.5, , 00000000007.27.9, , , , , ] 2021-02-11T09:51:09.702 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000003.25.9, , 00000000010.26.5, , 00000000007.27.9, , , , , ] 2021-02-11T09:51:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:09.700+0000 7fc93df32080 0 bucket checkpoint complete 2021-02-11T09:51:09.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:09.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:09.736 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:09.828 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:09.828 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:10.040+0000 7fabe0aaa080 1 bucket sync caught up with source: 2021-02-11T09:51:10.040 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000003.28.9, , 00000000003.15.9, , 00000000014.38.5, , , , , ] 2021-02-11T09:51:10.040 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000003.28.9, , 00000000003.15.9, , 00000000014.38.5, , , , , ] 2021-02-11T09:51:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:10.040+0000 7fabe0aaa080 0 bucket checkpoint complete 2021-02-11T09:51:10.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:10.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:10.070 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:10.116 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:10.117 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:10.335 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:10.335+0000 7f9e42c59080 1 bucket sync caught up with source: 2021-02-11T09:51:10.335 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000003.28.9, , 00000000003.15.9, , 00000000014.38.5, , , , , ] 2021-02-11T09:51:10.335 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000003.28.9, , 00000000003.15.9, , 00000000014.38.5, , , , , ] 2021-02-11T09:51:10.335 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:10.335+0000 7f9e42c59080 0 bucket checkpoint complete 2021-02-11T09:51:10.367 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-16 zones={test-zone2, test-zone1} 2021-02-11T09:51:11.448 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:51:11.456 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:11.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:11.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:11.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:11.456 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:51:11.462 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:11.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:11.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:11.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:11.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:11.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:11.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:11.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:11.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:11.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:11.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:11.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:12.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:12.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:12.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:12.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:12.446 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-16 zones={test-zone2, test-zone1} 2021-02-11T09:51:12.446 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-16 zones={test-zone3, test-zone1} 2021-02-11T09:51:12.760 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:51:12.766 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:12.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:12.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:12.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:12.766 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:51:12.770 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:12.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:12.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:12.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:12.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:12.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:12.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:12.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:12.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:13.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:13.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:13.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:13.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:13.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:13.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:13.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:13.717 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-16 zones={test-zone3, test-zone1} 2021-02-11T09:51:13.717 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-16 zones={test-zone3, test-zone2} 2021-02-11T09:51:14.596 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:51:14.603 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:14.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:14.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:14.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:14.603 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:51:14.609 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:14.609 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:14.609 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:14.609 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:14.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:14.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:14.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:14.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:14.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:14.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:15.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:15.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:15.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:15.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:15.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:15.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:15.568 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-16 zones={test-zone3, test-zone2} 2021-02-11T09:51:15.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:15.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:15.569 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:15.613 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:15.613 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:15.831 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:15.830+0000 7f7268e13080 1 bucket sync caught up with source: 2021-02-11T09:51:15.831 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000010.25.5, , 00000000003.26.9, , 00000000007.16.9, , , , , ] 2021-02-11T09:51:15.831 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000010.25.5, , 00000000003.26.9, , 00000000007.16.9, , , , , ] 2021-02-11T09:51:15.831 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:15.830+0000 7f7268e13080 0 bucket checkpoint complete 2021-02-11T09:51:15.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:15.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:15.864 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:15.912 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:15.912 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:16.117 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:16.115+0000 7f0fd393d080 1 bucket sync caught up with source: 2021-02-11T09:51:16.117 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000010.25.5, , 00000000003.26.9, , 00000000007.16.9, , , , , ] 2021-02-11T09:51:16.117 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000010.25.5, , 00000000003.26.9, , 00000000007.16.9, , , , , ] 2021-02-11T09:51:16.117 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:16.115+0000 7f0fd393d080 0 bucket checkpoint complete 2021-02-11T09:51:16.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:16.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:16.149 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:16.196 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:16.196 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:16.402 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:16.402+0000 7f0dc2c83080 1 bucket sync caught up with source: 2021-02-11T09:51:16.402 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000010.25.5, , 00000000003.26.9, , 00000000007.16.9, , , , , ] 2021-02-11T09:51:16.402 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000010.25.5, , 00000000003.26.9, , 00000000007.16.9, , , , , ] 2021-02-11T09:51:16.402 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:16.402+0000 7f0dc2c83080 0 bucket checkpoint complete 2021-02-11T09:51:16.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:16.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:16.434 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:16.481 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:16.481 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:16.715 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:16.711+0000 7fd33aaba080 1 bucket sync caught up with source: 2021-02-11T09:51:16.715 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000003.16.9, , 00000000010.48.5, , 00000000009.23.9, , , , , ] 2021-02-11T09:51:16.715 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000003.16.9, , 00000000010.48.5, , 00000000009.23.9, , , , , ] 2021-02-11T09:51:16.715 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:16.711+0000 7fd33aaba080 0 bucket checkpoint complete 2021-02-11T09:51:16.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:16.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:16.751 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:16.797 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:16.797 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:17.015 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:17.011+0000 7f5d267dc080 1 bucket sync caught up with source: 2021-02-11T09:51:17.015 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000003.16.9, , 00000000010.48.5, , 00000000009.23.9, , , , , ] 2021-02-11T09:51:17.015 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000003.16.9, , 00000000010.48.5, , 00000000009.23.9, , , , , ] 2021-02-11T09:51:17.015 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:17.011+0000 7f5d267dc080 0 bucket checkpoint complete 2021-02-11T09:51:17.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:17.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:17.048 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:17.093 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:17.094 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:17.304 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:17.303+0000 7fc0f9ec5080 1 bucket sync caught up with source: 2021-02-11T09:51:17.305 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000003.15.9, , 00000000003.40.9, , 00000000014.34.5, , , , , ] 2021-02-11T09:51:17.305 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000003.15.9, , 00000000003.40.9, , 00000000014.34.5, , , , , ] 2021-02-11T09:51:17.305 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:17.303+0000 7fc0f9ec5080 0 bucket checkpoint complete 2021-02-11T09:51:17.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:17.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:17.334 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:17.380 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:17.380 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:17.589 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:17.590+0000 7f77dc82b080 1 bucket sync caught up with source: 2021-02-11T09:51:17.589 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000003.15.9, , 00000000003.40.9, , 00000000014.34.5, , , , , ] 2021-02-11T09:51:17.589 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000003.15.9, , 00000000003.40.9, , 00000000014.34.5, , , , , ] 2021-02-11T09:51:17.589 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:17.590+0000 7f77dc82b080 0 bucket checkpoint complete 2021-02-11T09:51:17.618 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-17 zones={test-zone2, test-zone1} 2021-02-11T09:51:18.719 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:51:18.726 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:18.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:18.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:18.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:18.726 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:51:18.731 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:18.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:18.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:18.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:18.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:18.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:18.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:18.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:18.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:19.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:19.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:19.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:19.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:19.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:19.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:19.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:19.689 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-17 zones={test-zone2, test-zone1} 2021-02-11T09:51:19.689 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-17 zones={test-zone3, test-zone1} 2021-02-11T09:51:19.966 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:51:19.971 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:19.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:19.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:19.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:19.971 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:51:19.976 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:19.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:19.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:19.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:19.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:19.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:20.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:20.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:20.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:20.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:20.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:20.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:20.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:20.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:20.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:20.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:20.941 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-17 zones={test-zone3, test-zone1} 2021-02-11T09:51:20.941 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-17 zones={test-zone3, test-zone2} 2021-02-11T09:51:21.225 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:51:21.232 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:21.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:21.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:21.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:21.233 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:51:21.239 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:21.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:21.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:21.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:21.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:21.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:21.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:21.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:21.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:21.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:21.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:21.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:21.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:21.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:22.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:22.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:22.285 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-17 zones={test-zone3, test-zone2} 2021-02-11T09:51:22.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:22.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:22.286 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:22.330 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:22.331 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:22.538 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:22.538+0000 7ff9cf8cd080 1 bucket sync caught up with source: 2021-02-11T09:51:22.538 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000010.30.5, , 00000000003.7.9, , 00000000007.31.9, , , , , ] 2021-02-11T09:51:22.538 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000010.30.5, , 00000000003.7.9, , 00000000007.31.9, , , , , ] 2021-02-11T09:51:22.538 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:22.538+0000 7ff9cf8cd080 0 bucket checkpoint complete 2021-02-11T09:51:22.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:22.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:22.569 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:22.615 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:22.615 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:22.827 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:22.827+0000 7f4665bad080 1 bucket sync caught up with source: 2021-02-11T09:51:22.827 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000010.30.5, , 00000000003.7.9, , 00000000007.31.9, , , , , ] 2021-02-11T09:51:22.827 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000010.30.5, , 00000000003.7.9, , 00000000007.31.9, , , , , ] 2021-02-11T09:51:22.827 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:22.827+0000 7f4665bad080 0 bucket checkpoint complete 2021-02-11T09:51:22.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:22.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:22.858 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:22.906 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:22.906 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:23.136 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:23.134+0000 7fea0b9ac080 1 bucket sync caught up with source: 2021-02-11T09:51:23.137 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000010.30.5, , 00000000003.7.9, , 00000000007.31.9, , , , , ] 2021-02-11T09:51:23.137 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000010.30.5, , 00000000003.7.9, , 00000000007.31.9, , , , , ] 2021-02-11T09:51:23.137 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:23.134+0000 7fea0b9ac080 0 bucket checkpoint complete 2021-02-11T09:51:23.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:23.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:23.165 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:23.213 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:23.213 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:23.438 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:23.439+0000 7f49cc04d080 1 bucket sync caught up with source: 2021-02-11T09:51:23.438 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000003.23.9, , 00000000010.22.5, , 00000000007.57.9, , , , , ] 2021-02-11T09:51:23.438 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000003.23.9, , 00000000010.22.5, , 00000000007.57.9, , , , , ] 2021-02-11T09:51:23.439 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:23.439+0000 7f49cc04d080 0 bucket checkpoint complete 2021-02-11T09:51:23.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:23.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:23.475 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:23.522 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:23.522 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:23.740 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:23.739+0000 7f75a024e080 1 bucket sync caught up with source: 2021-02-11T09:51:23.740 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000003.23.9, , 00000000010.22.5, , 00000000007.57.9, , , , , ] 2021-02-11T09:51:23.740 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000003.23.9, , 00000000010.22.5, , 00000000007.57.9, , , , , ] 2021-02-11T09:51:23.740 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:23.739+0000 7f75a024e080 0 bucket checkpoint complete 2021-02-11T09:51:23.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:23.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:23.774 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:23.817 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:23.818 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:24.033 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:24.035+0000 7ff1b93d4080 1 bucket sync caught up with source: 2021-02-11T09:51:24.033 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000003.23.9, , 00000000003.12.9, , 00000000016.60.5, , , , , ] 2021-02-11T09:51:24.033 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000003.23.9, , 00000000003.12.9, , 00000000016.60.5, , , , , ] 2021-02-11T09:51:24.033 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:24.035+0000 7ff1b93d4080 0 bucket checkpoint complete 2021-02-11T09:51:24.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:24.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:24.063 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:24.111 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:24.111 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:24.338 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:24.338+0000 7fabcf0ba080 1 bucket sync caught up with source: 2021-02-11T09:51:24.338 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000003.23.9, , 00000000003.12.9, , 00000000016.60.5, , , , , ] 2021-02-11T09:51:24.338 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000003.23.9, , 00000000003.12.9, , 00000000016.60.5, , , , , ] 2021-02-11T09:51:24.338 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:24.338+0000 7fabcf0ba080 0 bucket checkpoint complete 2021-02-11T09:51:24.372 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-18 zones={test-zone2, test-zone1} 2021-02-11T09:51:25.524 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:51:25.532 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:25.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:25.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:25.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:25.532 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:51:25.539 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:25.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:25.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:25.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:25.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:25.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:25.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:25.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:25.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:25.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:25.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:26.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:26.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:26.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:26.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:26.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:26.518 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-18 zones={test-zone2, test-zone1} 2021-02-11T09:51:26.518 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-18 zones={test-zone3, test-zone1} 2021-02-11T09:51:26.872 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:51:26.878 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:26.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:26.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:26.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:26.878 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:51:26.883 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:26.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:26.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:26.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:26.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:26.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:26.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:27.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:27.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:27.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:27.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:27.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:27.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:35.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:35.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:36.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:36.117 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-18 zones={test-zone3, test-zone1} 2021-02-11T09:51:36.117 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-18 zones={test-zone3, test-zone2} 2021-02-11T09:51:36.685 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:51:36.693 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:36.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:36.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:36.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:36.694 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:51:36.699 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:36.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:36.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:36.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:36.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:36.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:36.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:36.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:36.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:37.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:37.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:37.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:37.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:37.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:37.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:37.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:37.658 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-18 zones={test-zone3, test-zone2} 2021-02-11T09:51:39.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:39.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:39.463 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:39.511 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:39.511 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:39.709 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:39.709+0000 7fa7166a1080 1 bucket sync caught up with source: 2021-02-11T09:51:39.709 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000011.40.5, , 00000000004.13.5, , 00000000010.29.5, , , , , ] 2021-02-11T09:51:39.709 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000011.40.5, , 00000000004.13.5, , 00000000010.29.5, , , , , ] 2021-02-11T09:51:39.709 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:39.709+0000 7fa7166a1080 0 bucket checkpoint complete 2021-02-11T09:51:39.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:39.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:39.737 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:39.792 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:39.792 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:39.995 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:39.994+0000 7ff64d6df080 1 bucket sync caught up with source: 2021-02-11T09:51:39.996 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000011.40.5, , 00000000004.13.5, , 00000000010.29.5, , , , , ] 2021-02-11T09:51:39.996 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000011.40.5, , 00000000004.13.5, , 00000000010.29.5, , , , , ] 2021-02-11T09:51:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:39.994+0000 7ff64d6df080 0 bucket checkpoint complete 2021-02-11T09:51:40.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:40.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:40.027 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:40.074 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:40.075 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:40.266 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:40.265+0000 7f2e9c843080 1 bucket sync caught up with source: 2021-02-11T09:51:40.266 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000011.40.5, , 00000000004.13.5, , 00000000010.29.5, , , , , ] 2021-02-11T09:51:40.266 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000011.40.5, , 00000000004.13.5, , 00000000010.29.5, , , , , ] 2021-02-11T09:51:40.267 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:40.265+0000 7f2e9c843080 0 bucket checkpoint complete 2021-02-11T09:51:40.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:40.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:40.295 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:40.345 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:40.345 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:40.550 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:40.550+0000 7f264a94c080 1 bucket sync caught up with source: 2021-02-11T09:51:40.550 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000004.28.5, , 00000000011.33.5, , 00000000008.30.5, , , , , ] 2021-02-11T09:51:40.551 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000004.28.5, , 00000000011.33.5, , 00000000008.30.5, , , , , ] 2021-02-11T09:51:40.551 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:40.550+0000 7f264a94c080 0 bucket checkpoint complete 2021-02-11T09:51:40.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:40.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:40.581 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:40.668 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:40.668 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:40.872 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:40.870+0000 7f09687ba080 1 bucket sync caught up with source: 2021-02-11T09:51:40.873 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000004.28.5, , 00000000011.33.5, , 00000000008.30.5, , , , , ] 2021-02-11T09:51:40.873 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000004.28.5, , 00000000011.33.5, , 00000000008.30.5, , , , , ] 2021-02-11T09:51:40.873 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:40.870+0000 7f09687ba080 0 bucket checkpoint complete 2021-02-11T09:51:40.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:40.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:40.908 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:40.956 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:40.956 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:41.189 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:41.186+0000 7f997dc3c080 1 bucket sync caught up with source: 2021-02-11T09:51:41.189 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000004.31.5, , 00000000004.18.5, , 00000000015.41.5, , , , , ] 2021-02-11T09:51:41.189 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000004.31.5, , 00000000004.18.5, , 00000000015.41.5, , , , , ] 2021-02-11T09:51:41.189 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:41.186+0000 7f997dc3c080 0 bucket checkpoint complete 2021-02-11T09:51:41.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:41.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:41.219 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:41.267 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:41.267 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:41.473 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:41.469+0000 7f9d9ac6e080 1 bucket sync caught up with source: 2021-02-11T09:51:41.473 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000004.31.5, , 00000000004.18.5, , 00000000015.41.5, , , , , ] 2021-02-11T09:51:41.473 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000004.31.5, , 00000000004.18.5, , 00000000015.41.5, , , , , ] 2021-02-11T09:51:41.473 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:41.469+0000 7f9d9ac6e080 0 bucket checkpoint complete 2021-02-11T09:51:41.505 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-16 zones={test-zone2, test-zone1} 2021-02-11T09:51:42.349 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:51:42.356 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:42.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:42.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:42.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:42.356 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:51:42.362 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:42.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:42.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:42.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:42.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:42.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:42.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:42.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:42.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:42.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:42.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:42.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:42.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:43.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:43.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:43.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:43.389 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-16 zones={test-zone2, test-zone1} 2021-02-11T09:51:43.389 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-16 zones={test-zone3, test-zone1} 2021-02-11T09:51:44.207 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:51:44.212 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:44.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:44.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:44.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:44.212 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:51:44.216 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:44.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:44.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:44.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:44.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:44.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:44.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:44.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:44.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:44.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:44.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:44.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:44.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:44.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:44.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:45.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:45.185 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-16 zones={test-zone3, test-zone1} 2021-02-11T09:51:45.185 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-16 zones={test-zone3, test-zone2} 2021-02-11T09:51:46.101 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:51:46.110 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:46.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:46.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:46.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:46.111 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:51:46.116 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:46.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:46.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:46.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:46.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:46.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:46.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:46.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:46.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:46.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:46.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:46.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:46.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:46.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:46.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:46.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:47.094 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-16 zones={test-zone3, test-zone2} 2021-02-11T09:51:47.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:47.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:47.095 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:47.141 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:47.141 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:47.366 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:47.365+0000 7f19ed714080 1 bucket sync caught up with source: 2021-02-11T09:51:47.366 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000011.31.5, , 00000000004.35.5, , 00000000008.19.5, , , , , ] 2021-02-11T09:51:47.366 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000011.31.5, , 00000000004.35.5, , 00000000008.19.5, , , , , ] 2021-02-11T09:51:47.366 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:47.365+0000 7f19ed714080 0 bucket checkpoint complete 2021-02-11T09:51:47.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:47.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:47.402 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:47.450 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:47.451 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:47.678 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:47.677+0000 7faf09117080 1 bucket sync caught up with source: 2021-02-11T09:51:47.679 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000011.31.5, , 00000000004.35.5, , 00000000008.19.5, , , , , ] 2021-02-11T09:51:47.679 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000011.31.5, , 00000000004.35.5, , 00000000008.19.5, , , , , ] 2021-02-11T09:51:47.679 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:47.677+0000 7faf09117080 0 bucket checkpoint complete 2021-02-11T09:51:47.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:47.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:47.715 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:47.761 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:47.762 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:47.969 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:47.969+0000 7fa711e51080 1 bucket sync caught up with source: 2021-02-11T09:51:47.970 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000011.31.5, , 00000000004.35.5, , 00000000008.19.5, , , , , ] 2021-02-11T09:51:47.970 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000011.31.5, , 00000000004.35.5, , 00000000008.19.5, , , , , ] 2021-02-11T09:51:47.970 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:47.969+0000 7fa711e51080 0 bucket checkpoint complete 2021-02-11T09:51:48.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:48.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:48.006 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:48.054 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:48.055 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:48.272 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:48.269+0000 7f8b08f84080 1 bucket sync caught up with source: 2021-02-11T09:51:48.272 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000004.19.5, , 00000000011.60.5, , 00000000010.26.5, , , , , ] 2021-02-11T09:51:48.272 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000004.19.5, , 00000000011.60.5, , 00000000010.26.5, , , , , ] 2021-02-11T09:51:48.272 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:48.269+0000 7f8b08f84080 0 bucket checkpoint complete 2021-02-11T09:51:48.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:48.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:48.308 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:48.354 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:48.354 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:48.592 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:48.589+0000 7fc696e3a080 1 bucket sync caught up with source: 2021-02-11T09:51:48.592 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000004.19.5, , 00000000011.60.5, , 00000000010.26.5, , , , , ] 2021-02-11T09:51:48.592 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000004.19.5, , 00000000011.60.5, , 00000000010.26.5, , , , , ] 2021-02-11T09:51:48.592 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:48.589+0000 7fc696e3a080 0 bucket checkpoint complete 2021-02-11T09:51:48.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:48.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:48.627 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:48.671 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:48.671 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:48.885 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:48.881+0000 7fcc05463080 1 bucket sync caught up with source: 2021-02-11T09:51:48.885 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000004.18.5, , 00000000004.63.5, , 00000000015.37.5, , , , , ] 2021-02-11T09:51:48.885 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000004.18.5, , 00000000004.63.5, , 00000000015.37.5, , , , , ] 2021-02-11T09:51:48.885 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:48.881+0000 7fcc05463080 0 bucket checkpoint complete 2021-02-11T09:51:48.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:48.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:48.916 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:48.964 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:48.964 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:49.186 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:49.185+0000 7fbb84fe6080 1 bucket sync caught up with source: 2021-02-11T09:51:49.187 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000004.18.5, , 00000000004.63.5, , 00000000015.37.5, , , , , ] 2021-02-11T09:51:49.187 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000004.18.5, , 00000000004.63.5, , 00000000015.37.5, , , , , ] 2021-02-11T09:51:49.187 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:49.185+0000 7fbb84fe6080 0 bucket checkpoint complete 2021-02-11T09:51:49.223 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-17 zones={test-zone2, test-zone1} 2021-02-11T09:51:49.315 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:51:49.321 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:49.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:49.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:49.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:49.322 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:51:49.328 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:49.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:49.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:49.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:49.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:49.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:49.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:49.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:49.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:49.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:49.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:49.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:49.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:50.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:50.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:50.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:50.294 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-17 zones={test-zone2, test-zone1} 2021-02-11T09:51:50.294 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-17 zones={test-zone3, test-zone1} 2021-02-11T09:51:50.376 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:51:50.381 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:50.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:50.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:50.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:50.381 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:51:50.385 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:50.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:50.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:50.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:50.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:50.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:50.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:50.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:50.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:50.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:50.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:50.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:50.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:51.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:51.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:51.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:51.353 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-17 zones={test-zone3, test-zone1} 2021-02-11T09:51:51.353 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-17 zones={test-zone3, test-zone2} 2021-02-11T09:51:51.919 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:51:51.926 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:51.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:51.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:51.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:51.926 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:51:51.932 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:51.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:51.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:51.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:51.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:51.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:52.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:52.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:52.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:52.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:52.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:52.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:52.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:52.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:52.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:52.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:52.904 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-17 zones={test-zone3, test-zone2} 2021-02-11T09:51:52.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:52.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:52.905 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:52.949 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:52.949 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:53.154 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:53.152+0000 7f99c95d2080 1 bucket sync caught up with source: 2021-02-11T09:51:53.154 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000011.36.5, , 00000000004.10.5, , 00000000008.37.5, , , , , ] 2021-02-11T09:51:53.154 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000011.36.5, , 00000000004.10.5, , 00000000008.37.5, , , , , ] 2021-02-11T09:51:53.154 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:53.152+0000 7f99c95d2080 0 bucket checkpoint complete 2021-02-11T09:51:53.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:53.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:53.190 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:53.240 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:53.240 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:53.457 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:53.453+0000 7f6535322080 1 bucket sync caught up with source: 2021-02-11T09:51:53.457 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000011.36.5, , 00000000004.10.5, , 00000000008.37.5, , , , , ] 2021-02-11T09:51:53.457 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000011.36.5, , 00000000004.10.5, , 00000000008.37.5, , , , , ] 2021-02-11T09:51:53.457 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:53.453+0000 7f6535322080 0 bucket checkpoint complete 2021-02-11T09:51:53.487 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:53.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:53.487 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:53.534 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:53.534 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:53.740 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:53.736+0000 7f417971a080 1 bucket sync caught up with source: 2021-02-11T09:51:53.741 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000011.36.5, , 00000000004.10.5, , 00000000008.37.5, , , , , ] 2021-02-11T09:51:53.741 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000011.36.5, , 00000000004.10.5, , 00000000008.37.5, , , , , ] 2021-02-11T09:51:53.741 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:53.736+0000 7f417971a080 0 bucket checkpoint complete 2021-02-11T09:51:53.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:53.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:53.772 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:53.813 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:53.814 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:54.021 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:54.017+0000 7f80188de080 1 bucket sync caught up with source: 2021-02-11T09:51:54.022 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000004.26.5, , 00000000011.28.5, , 00000000008.62.5, , , , , ] 2021-02-11T09:51:54.022 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000004.26.5, , 00000000011.28.5, , 00000000008.62.5, , , , , ] 2021-02-11T09:51:54.022 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:54.017+0000 7f80188de080 0 bucket checkpoint complete 2021-02-11T09:51:54.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:54.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:54.059 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:54.148 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:54.148 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:54.376 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:54.373+0000 7f7b87172080 1 bucket sync caught up with source: 2021-02-11T09:51:54.376 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000004.26.5, , 00000000011.28.5, , 00000000008.62.5, , , , , ] 2021-02-11T09:51:54.377 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000004.26.5, , 00000000011.28.5, , 00000000008.62.5, , , , , ] 2021-02-11T09:51:54.377 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:54.373+0000 7f7b87172080 0 bucket checkpoint complete 2021-02-11T09:51:54.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:54.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:54.412 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:54.456 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:54.456 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:54.707 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:54.705+0000 7feb640a2080 1 bucket sync caught up with source: 2021-02-11T09:51:54.707 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000004.26.5, , 00000000004.15.5, , 00000000017.65.5, , , , , ] 2021-02-11T09:51:54.707 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000004.26.5, , 00000000004.15.5, , 00000000017.65.5, , , , , ] 2021-02-11T09:51:54.708 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:51:54.705+0000 7feb640a2080 0 bucket checkpoint complete 2021-02-11T09:51:54.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:54.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:51:54.739 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:51:54.788 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:54.788 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:54.992 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:54.988+0000 7f70b6bbc080 1 bucket sync caught up with source: 2021-02-11T09:51:54.992 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000004.26.5, , 00000000004.15.5, , 00000000017.65.5, , , , , ] 2021-02-11T09:51:54.992 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000004.26.5, , 00000000004.15.5, , 00000000017.65.5, , , , , ] 2021-02-11T09:51:54.992 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:51:54.988+0000 7f70b6bbc080 0 bucket checkpoint complete 2021-02-11T09:51:55.027 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-18 zones={test-zone2, test-zone1} 2021-02-11T09:51:56.851 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:51:56.859 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:56.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:56.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:56.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:56.860 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:51:56.868 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:51:56.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:51:56.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:51:56.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:51:56.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:56.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:56.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:51:57.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:57.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:57.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:51:57.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:57.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:57.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:51:57.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:57.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:57.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:51:57.869 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-18 zones={test-zone2, test-zone1} 2021-02-11T09:51:57.869 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-18 zones={test-zone3, test-zone1} 2021-02-11T09:51:58.699 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:52:00.696 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:52:00.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:52:00.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:52:00.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:52:00.696 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:52:01.702 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:52:01.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:52:01.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:52:01.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:52:02.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:02.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:02.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:02.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:52:02.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:52:02.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:52:03.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:52:03.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:52:03.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:52:03.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:52:03.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:52:03.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:52:03.600 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-18 zones={test-zone3, test-zone1} 2021-02-11T09:52:03.600 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-18 zones={test-zone3, test-zone2} 2021-02-11T09:52:03.773 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:52:03.781 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:52:03.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:52:03.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:52:03.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:52:03.781 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:52:03.788 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:52:03.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:52:03.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:52:03.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:52:03.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:03.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:03.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:03.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:52:03.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:52:04.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:52:04.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:52:04.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:52:04.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:52:04.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:52:04.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:52:04.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:52:04.765 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-18 zones={test-zone3, test-zone2} 2021-02-11T09:52:04.766 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2021-02-11T09:52:05.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:05.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:05.516 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:52:05.549 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:05.550 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:05.719 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f2716509-104d-423b-ba5c-db1bd74642ab 2021-02-11T09:52:05.756 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613036985.070217_101.1","last_update":"2021-02-11T09:49:45.070217Z"},{"marker":"1_1613037125.514387_206.1","last_update":"2021-02-11T09:52:05.514387Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:52:05.757 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:52:05.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:05.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:05.757 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:52:05.792 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:05.792 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:06.006 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037125.514387_206.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:05.514387Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:52:06.006 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:52:06.007 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:52:06.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:06.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:06.007 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:52:06.048 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:06.048 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:06.280 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037125.514387_206.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:05.514387Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:52:06.280 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:52:06.281 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:52:06.281 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:06.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:06.281 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:52:06.324 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:06.324 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:06.536 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037125.514387_206.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:05.514387Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:52:06.536 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:52:07.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:07.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:07.323 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:52:07.372 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:07.372 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:07.583 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:52:07.579+0000 7f856caf2080 1 bucket sync caught up with source: 2021-02-11T09:52:07.583 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , 00000000002.9.10, , , , , ] 2021-02-11T09:52:07.583 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , 00000000002.9.10, , , , , ] 2021-02-11T09:52:07.583 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:52:07.579+0000 7f856caf2080 0 bucket checkpoint complete 2021-02-11T09:52:07.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:07.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:07.613 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:52:07.665 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:07.665 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:07.854 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:52:07.848+0000 7f844000f080 1 waiting for incremental sync to catch up: 2021-02-11T09:52:07.854 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000004.13.9, , , , , ] 2021-02-11T09:52:07.854 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000005.15.9, , , , , ] 2021-02-11T09:52:12.857 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:52:12.852+0000 7f844000f080 1 bucket sync caught up with source: 2021-02-11T09:52:12.857 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000005.15.9, , , , , ] 2021-02-11T09:52:12.858 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000005.15.9, , , , , ] 2021-02-11T09:52:12.858 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:52:12.852+0000 7f844000f080 0 bucket checkpoint complete 2021-02-11T09:52:12.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:12.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:12.891 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:52:12.938 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:12.938 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:13.146 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:52:13.143+0000 7fec2b13d080 1 bucket sync caught up with source: 2021-02-11T09:52:13.146 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , 00000000005.15.9, , , , , ] 2021-02-11T09:52:13.146 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , 00000000005.15.9, , , , , ] 2021-02-11T09:52:13.146 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:52:13.143+0000 7fec2b13d080 0 bucket checkpoint complete 2021-02-11T09:52:13.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:13.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:13.177 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:52:13.224 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:13.224 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:13.456 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:52:13.452+0000 7f8cb7839080 1 bucket sync caught up with source: 2021-02-11T09:52:13.456 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000004.19.9, , , , , ] 2021-02-11T09:52:13.456 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000004.19.9, , , , , ] 2021-02-11T09:52:13.456 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:52:13.452+0000 7f8cb7839080 0 bucket checkpoint complete 2021-02-11T09:52:13.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:13.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:13.490 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:52:13.534 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:13.534 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:13.762 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:52:13.756+0000 7ff09ecb9080 1 bucket sync caught up with source: 2021-02-11T09:52:13.762 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000004.19.9, , , , , ] 2021-02-11T09:52:13.762 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000004.19.9, , , , , ] 2021-02-11T09:52:13.763 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:52:13.756+0000 7ff09ecb9080 0 bucket checkpoint complete 2021-02-11T09:52:13.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:13.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:13.795 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:52:13.892 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:13.892 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:14.116 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:52:14.112+0000 7fbe19d49080 1 bucket sync caught up with source: 2021-02-11T09:52:14.116 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000004.18.9, , , , , ] 2021-02-11T09:52:14.116 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000004.18.9, , , , , ] 2021-02-11T09:52:14.116 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:52:14.112+0000 7fbe19d49080 0 bucket checkpoint complete 2021-02-11T09:52:14.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:14.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:14.148 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:52:14.197 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:14.197 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:14.414 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:52:14.411+0000 7fb80f3ee080 1 bucket sync caught up with source: 2021-02-11T09:52:14.415 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , 00000000004.18.9, , , , , ] 2021-02-11T09:52:14.415 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , 00000000004.18.9, , , , , ] 2021-02-11T09:52:14.415 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:52:14.411+0000 7fb80f3ee080 0 bucket checkpoint complete 2021-02-11T09:52:14.451 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-19 zones={test-zone2, test-zone1} 2021-02-11T09:52:15.340 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:52:15.348 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:52:15.348 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:52:15.394 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:52:15.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:15.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:15.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:15.594 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-19 zones={test-zone2, test-zone1} 2021-02-11T09:52:15.594 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-19 zones={test-zone3, test-zone1} 2021-02-11T09:52:15.756 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:52:15.806 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:52:15.807 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:52:15.813 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:52:15.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:15.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:15.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:16.053 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-19 zones={test-zone3, test-zone1} 2021-02-11T09:52:16.054 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-19 zones={test-zone3, test-zone2} 2021-02-11T09:52:16.064 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:52:16.115 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:52:16.115 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:52:16.121 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:52:16.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:16.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:16.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:16.318 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-19 zones={test-zone3, test-zone2} 2021-02-11T09:52:17.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:17.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:17.823 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:52:17.875 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:17.875 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:18.057 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:52:18.055+0000 7f51a2621080 1 waiting for incremental sync to catch up: 2021-02-11T09:52:18.057 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , 00000000005.15.9, , , 00000000045.123.11, , ] 2021-02-11T09:52:18.057 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , 00000000005.15.9, , , 00000000047.129.8, , ] 2021-02-11T09:52:23.061 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:52:23.058+0000 7f51a2621080 1 bucket sync caught up with source: 2021-02-11T09:52:23.061 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , 00000000005.15.9, , , 00000000049.133.9, , ] 2021-02-11T09:52:23.061 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , 00000000005.15.9, , , 00000000047.129.8, , ] 2021-02-11T09:52:23.061 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:52:23.058+0000 7f51a2621080 0 bucket checkpoint complete 2021-02-11T09:52:23.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:23.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:23.095 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:52:23.143 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:23.143 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:23.358 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:52:23.352+0000 7f17affeb080 1 bucket sync caught up with source: 2021-02-11T09:52:23.359 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000005.15.9, , , 00000000049.133.9, , ] 2021-02-11T09:52:23.359 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000005.15.9, , , 00000000049.133.9, , ] 2021-02-11T09:52:23.359 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:52:23.352+0000 7f17affeb080 0 bucket checkpoint complete 2021-02-11T09:52:23.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:23.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:23.389 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:52:23.436 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:23.436 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:23.651 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:52:23.650+0000 7f937b694080 1 bucket sync caught up with source: 2021-02-11T09:52:23.651 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , 00000000005.15.9, , , 00000000049.133.9, , ] 2021-02-11T09:52:23.651 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , 00000000005.15.9, , , 00000000049.133.9, , ] 2021-02-11T09:52:23.651 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:52:23.650+0000 7f937b694080 0 bucket checkpoint complete 2021-02-11T09:52:23.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:23.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:23.686 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:52:23.733 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:23.733 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:23.956 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:52:23.952+0000 7f2e1e654080 1 bucket sync caught up with source: 2021-02-11T09:52:23.956 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000052.133.9, , ] 2021-02-11T09:52:23.957 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000052.133.9, , ] 2021-02-11T09:52:23.957 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:52:23.952+0000 7f2e1e654080 0 bucket checkpoint complete 2021-02-11T09:52:23.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:23.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:23.992 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:52:24.035 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:24.035 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:24.248 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:52:24.244+0000 7f6410c55080 1 bucket sync caught up with source: 2021-02-11T09:52:24.249 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000052.133.9, , ] 2021-02-11T09:52:24.249 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000052.133.9, , ] 2021-02-11T09:52:24.249 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:52:24.244+0000 7f6410c55080 0 bucket checkpoint complete 2021-02-11T09:52:24.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:24.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:24.283 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:52:24.372 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:24.372 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:24.593 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:52:24.588+0000 7f9eb15b0080 1 bucket sync caught up with source: 2021-02-11T09:52:24.594 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000004.18.9, , , 00000000062.147.9, , ] 2021-02-11T09:52:24.594 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000004.18.9, , , 00000000062.147.9, , ] 2021-02-11T09:52:24.594 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:52:24.588+0000 7f9eb15b0080 0 bucket checkpoint complete 2021-02-11T09:52:24.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:24.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:52:24.626 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:52:24.670 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:24.670 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:24.878 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:52:24.874+0000 7ff84d78f080 1 bucket sync caught up with source: 2021-02-11T09:52:24.879 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , 00000000004.18.9, , , 00000000062.147.9, , ] 2021-02-11T09:52:24.879 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , 00000000004.18.9, , , 00000000062.147.9, , ] 2021-02-11T09:52:24.879 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:52:24.874+0000 7ff84d78f080 0 bucket checkpoint complete 2021-02-11T09:52:24.912 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-19 zones={test-zone2, test-zone1} 2021-02-11T09:52:25.376 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:52:25.387 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:52:25.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.389 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:52:25.398 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:52:25.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:25.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:25.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:25.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:25.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:25.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:25.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:25.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:25.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:26.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:26.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:26.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:26.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:26.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:26.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:26.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:26.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:26.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:26.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:26.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:27.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:27.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:27.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:27.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:27.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:27.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:27.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:27.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:27.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:27.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:28.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:28.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:28.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:28.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:28.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:28.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:28.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:28.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:28.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:28.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:28.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:29.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:29.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:34.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:34.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:34.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:34.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:34.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:34.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:34.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:34.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:34.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:34.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:35.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:35.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:35.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:35.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:35.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:35.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:35.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:35.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:35.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:36.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:36.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:36.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:36.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:36.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:36.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:36.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:36.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:36.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:36.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:37.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:37.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:37.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:37.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:37.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:37.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:37.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:37.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:37.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:38.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:38.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:38.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:38.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:38.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:38.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:38.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:38.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:38.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:39.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:39.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:39.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:39.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:39.460 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-19 zones={test-zone2, test-zone1} 2021-02-11T09:52:39.461 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-19 zones={test-zone3, test-zone1} 2021-02-11T09:52:40.257 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:52:40.264 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:52:40.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.266 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:52:40.272 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:52:40.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:40.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:40.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:40.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:40.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:40.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:40.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:40.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:40.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:40.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:40.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:41.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:41.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:41.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:41.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:41.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:41.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:41.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:41.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:41.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:41.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:41.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:42.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:42.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:42.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:42.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:42.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:42.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:42.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:42.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:42.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:42.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:42.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:43.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:43.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:43.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:43.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:43.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:43.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:43.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:43.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:43.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:43.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:44.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:44.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:44.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:44.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:44.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:44.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:44.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:44.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:44.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:45.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:45.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:45.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:45.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:45.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:45.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:45.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:45.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:45.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:45.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:46.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:46.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:46.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:46.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:46.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:46.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:46.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:46.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:46.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:46.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:46.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:47.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:47.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:47.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:47.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:47.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:47.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:47.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:47.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:47.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:47.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:47.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:48.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:48.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:48.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:48.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:48.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:48.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:48.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:48.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:48.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:48.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:49.038 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-19 zones={test-zone3, test-zone1} 2021-02-11T09:52:49.038 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-19 zones={test-zone3, test-zone2} 2021-02-11T09:52:49.650 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:52:49.661 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:52:49.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.663 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:52:49.672 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:52:49.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:52:49.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:49.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:49.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:52:49.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:49.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:50.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:50.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:50.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:50.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:50.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:50.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:50.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:50.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:50.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:50.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:50.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:51.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:51.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:51.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:51.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:51.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:51.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:51.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:51.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:51.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:51.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:52.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:52.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:52.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:52.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:52.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:52.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:52.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:52.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:52.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:52.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:52.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:53.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:53.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:53.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:53.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:53.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:53.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:53.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:53.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:53.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:53.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:53.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:54.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:54.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:54.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:54.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:54.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:54.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:54.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:54.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:54.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:54.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:54.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:55.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:55.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:55.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:55.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:55.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:55.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:55.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:55.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:55.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:55.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:56.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:56.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:56.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:56.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:56.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:56.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:56.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:56.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:56.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:56.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:56.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:57.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:57.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:57.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:57.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:57.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:57.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:57.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:57.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:57.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:57.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:58.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:58.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:58.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:52:58.283 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-19 zones={test-zone3, test-zone2} 2021-02-11T09:52:58.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:58.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:58.284 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:52:58.325 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:58.325 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:58.590 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037144.076572_222.1","last_update":"2021-02-11T09:52:24.076572Z"},{"marker":"1_1613037144.076572_157.1","last_update":"2021-02-11T09:52:24.076572Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1613037100.066735_115.3","last_update":"2021-02-11T09:51:40.066735Z"}] 2021-02-11T09:52:58.591 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T09:52:58.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:58.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:58.591 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:52:58.631 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:58.631 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:58.870 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037144.076572_222.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.076572Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037144.076572_157.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.076572Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037100.066735_115.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:51:40.066735Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:52:58.870 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T09:52:58.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:58.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:58.871 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:52:58.912 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:58.913 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:59.147 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037144.076572_222.1","last_update":"2021-02-11T09:52:24.076572Z"},{"marker":"1_1613037144.076572_157.1","last_update":"2021-02-11T09:52:24.076572Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1613037100.066735_115.3","last_update":"2021-02-11T09:51:40.066735Z"}] 2021-02-11T09:52:59.147 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T09:52:59.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:59.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:59.148 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:52:59.231 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:59.231 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:59.472 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7045610624405686142},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037144.076572_222.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.076572Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037144.076572_157.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.076572Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037100.066735_115.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:51:40.066735Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:52:59.472 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T09:52:59.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:59.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:59.473 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:52:59.552 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:59.553 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:59.779 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037144.076572_222.1","last_update":"2021-02-11T09:52:24.076572Z"},{"marker":"1_1613037144.076572_157.1","last_update":"2021-02-11T09:52:24.076572Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1613037100.066735_115.3","last_update":"2021-02-11T09:51:40.066735Z"}] 2021-02-11T09:52:59.779 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T09:52:59.779 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:59.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:59.780 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:52:59.825 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:59.825 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:00.055 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1531294025241915737},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037144.076572_222.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.076572Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037144.076572_157.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.076572Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037100.066735_115.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:51:40.066735Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:53:00.055 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T09:53:00.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:00.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:00.055 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:00.093 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:00.094 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:00.315 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1613037144.042530_205.1","last_update":"2021-02-11T09:52:24.042530Z"},{"marker":"1_1613037144.042530_160.1","last_update":"2021-02-11T09:52:24.042530Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1613037100.031804_163.3","last_update":"2021-02-11T09:51:40.031804Z"}] 2021-02-11T09:53:03.934 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T09:53:03.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:03.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:03.935 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:04.229 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3492294689851219663},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037144.042530_205.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.042530Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037144.042530_160.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.042530Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037100.031804_163.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:51:40.031804Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:53:04.229 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T09:53:04.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:04.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:04.230 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:04.270 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:04.270 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:04.509 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1613037144.042530_205.1","last_update":"2021-02-11T09:52:24.042530Z"},{"marker":"1_1613037144.042530_160.1","last_update":"2021-02-11T09:52:24.042530Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1613037100.031804_163.3","last_update":"2021-02-11T09:51:40.031804Z"}] 2021-02-11T09:53:04.509 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T09:53:04.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:04.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:04.510 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:04.551 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:04.551 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:04.784 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7442380677805818242},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037144.042530_205.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.042530Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037144.042530_160.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.042530Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037100.031804_163.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:51:40.031804Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:53:04.784 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T09:53:04.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:04.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:04.785 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:04.865 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:04.865 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:05.101 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037144.107937_207.1","last_update":"2021-02-11T09:52:24.107937Z"},{"marker":"1_1613037144.107937_91.1","last_update":"2021-02-11T09:52:24.107937Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1613037100.096926_105.3","last_update":"2021-02-11T09:51:40.096926Z"}] 2021-02-11T09:53:05.102 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T09:53:05.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:05.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:05.102 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:05.187 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:05.187 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:05.426 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14314976040552073418},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037144.107937_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.107937Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037144.107937_91.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.107937Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037100.096926_105.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:51:40.096926Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:53:05.427 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T09:53:05.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:05.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:05.427 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:05.466 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:05.467 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:05.719 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037144.107937_207.1","last_update":"2021-02-11T09:52:24.107937Z"},{"marker":"1_1613037144.107937_91.1","last_update":"2021-02-11T09:52:24.107937Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1613037100.096926_105.3","last_update":"2021-02-11T09:51:40.096926Z"}] 2021-02-11T09:53:05.719 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T09:53:05.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:05.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:05.719 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:05.762 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:05.763 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:06.009 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12589397483957797345},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037144.107937_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.107937Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037144.107937_91.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.107937Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037100.096926_105.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:51:40.096926Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:53:06.010 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T09:53:06.010 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2021-02-11T09:53:06.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:06.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:06.373 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:06.409 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:06.409 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:06.618 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f2716509-104d-423b-ba5c-db1bd74642ab 2021-02-11T09:53:06.651 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613036985.070217_101.1","last_update":"2021-02-11T09:49:45.070217Z"},{"marker":"1_1613037186.368386_210.1","last_update":"2021-02-11T09:53:06.368386Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:53:06.651 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:53:06.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:06.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:06.652 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:06.695 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:06.695 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:06.922 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037186.368386_210.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:06.368386Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:06.923 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:53:06.923 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:53:06.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:06.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:06.923 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:06.965 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:07.210 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037186.368386_210.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:06.368386Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:07.210 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:53:07.210 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:53:07.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:07.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:07.211 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:07.254 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:07.254 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:07.485 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037186.368386_210.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:06.368386Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:07.486 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:53:08.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:08.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:08.267 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:08.316 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:08.317 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:08.539 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:08.533+0000 7f5d95679080 1 bucket sync caught up with source: 2021-02-11T09:53:08.539 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000001.17.5, , ] 2021-02-11T09:53:08.539 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000001.17.5, , ] 2021-02-11T09:53:08.539 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:08.533+0000 7f5d95679080 0 bucket checkpoint complete 2021-02-11T09:53:08.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:08.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:08.572 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:08.623 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:08.817 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:08.815+0000 7f7ab4603080 1 bucket sync caught up with source: 2021-02-11T09:53:08.817 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000001.17.5, , ] 2021-02-11T09:53:08.817 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000001.17.5, , ] 2021-02-11T09:53:08.817 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:08.815+0000 7f7ab4603080 0 bucket checkpoint complete 2021-02-11T09:53:08.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:08.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:08.859 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:08.908 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:08.908 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:09.121 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:09.117+0000 7f1df4ebf080 1 bucket sync caught up with source: 2021-02-11T09:53:09.122 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000001.17.5, , ] 2021-02-11T09:53:09.122 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000001.17.5, , ] 2021-02-11T09:53:09.122 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:09.117+0000 7f1df4ebf080 0 bucket checkpoint complete 2021-02-11T09:53:09.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:09.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:09.155 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:09.203 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:09.434 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:09.431+0000 7ff937994080 1 bucket sync caught up with source: 2021-02-11T09:53:09.435 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000001.37.5, , ] 2021-02-11T09:53:09.435 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000001.37.5, , ] 2021-02-11T09:53:09.435 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:09.431+0000 7ff937994080 0 bucket checkpoint complete 2021-02-11T09:53:09.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:09.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:09.472 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:09.520 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:09.520 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:09.742 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:09.739+0000 7f499569d080 1 bucket sync caught up with source: 2021-02-11T09:53:09.742 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000001.37.5, , ] 2021-02-11T09:53:09.743 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000001.37.5, , ] 2021-02-11T09:53:09.743 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:09.739+0000 7f499569d080 0 bucket checkpoint complete 2021-02-11T09:53:09.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:09.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:09.778 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:09.822 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:09.822 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:10.050 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:10.047+0000 7fdb7f930080 1 bucket sync caught up with source: 2021-02-11T09:53:10.050 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000001.22.5, , ] 2021-02-11T09:53:10.051 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000001.22.5, , ] 2021-02-11T09:53:10.051 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:10.047+0000 7fdb7f930080 0 bucket checkpoint complete 2021-02-11T09:53:10.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:10.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:10.080 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:10.127 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:10.127 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:10.349 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:10.345+0000 7fe7eb66b080 1 bucket sync caught up with source: 2021-02-11T09:53:10.349 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000001.22.5, , ] 2021-02-11T09:53:10.349 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000001.22.5, , ] 2021-02-11T09:53:10.349 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:10.345+0000 7fe7eb66b080 0 bucket checkpoint complete 2021-02-11T09:53:10.375 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-20 zones={test-zone2, test-zone1} 2021-02-11T09:53:11.541 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:11.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:11.548 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:11.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:11.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:11.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:11.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:11.793 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-20 zones={test-zone2, test-zone1} 2021-02-11T09:53:11.793 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-20 zones={test-zone3, test-zone1} 2021-02-11T09:53:12.705 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:12.751 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:12.751 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:12.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:12.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:12.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:12.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:12.949 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-20 zones={test-zone3, test-zone1} 2021-02-11T09:53:12.949 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-20 zones={test-zone3, test-zone2} 2021-02-11T09:53:13.680 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:13.688 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:13.688 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:13.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:13.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:13.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:13.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:13.929 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-20 zones={test-zone3, test-zone2} 2021-02-11T09:53:13.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:13.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:13.947 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:13.980 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:13.980 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:14.176 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f2716509-104d-423b-ba5c-db1bd74642ab 2021-02-11T09:53:14.216 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613036985.070217_101.1","last_update":"2021-02-11T09:49:45.070217Z"},{"marker":"1_1613037193.942339_212.1","last_update":"2021-02-11T09:53:13.942339Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:53:14.216 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:53:14.217 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:14.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:14.217 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:14.250 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:14.251 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:14.479 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037193.942339_212.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:13.942339Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:14.480 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:53:14.480 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:53:14.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:14.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:14.480 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:14.521 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:14.521 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:14.763 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037193.942339_212.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:13.942339Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:14.763 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:53:14.763 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:53:14.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:14.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:14.764 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:14.805 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:14.805 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:15.033 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037193.942339_212.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:13.942339Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:15.034 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:53:15.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:15.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:15.111 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:15.162 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:15.162 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:15.362 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:15.357+0000 7f7a9b33c080 1 bucket sync caught up with source: 2021-02-11T09:53:15.362 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000003.20.12, , ] 2021-02-11T09:53:15.362 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000003.20.12, , ] 2021-02-11T09:53:15.362 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:15.357+0000 7f7a9b33c080 0 bucket checkpoint complete 2021-02-11T09:53:15.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:15.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:15.389 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:15.437 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:15.437 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:15.655+0000 7f68fc875080 1 bucket sync caught up with source: 2021-02-11T09:53:15.658 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000003.20.12, , ] 2021-02-11T09:53:15.658 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000003.20.12, , ] 2021-02-11T09:53:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:15.655+0000 7f68fc875080 0 bucket checkpoint complete 2021-02-11T09:53:15.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:15.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:15.689 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:15.739 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:15.739 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:15.938 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:15.933+0000 7fe71c4fe080 1 bucket sync caught up with source: 2021-02-11T09:53:15.938 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000003.20.12, , ] 2021-02-11T09:53:15.938 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000003.20.12, , ] 2021-02-11T09:53:15.938 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:15.933+0000 7fe71c4fe080 0 bucket checkpoint complete 2021-02-11T09:53:15.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:15.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:15.974 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:16.025 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:16.026 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:16.259 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:16.259+0000 7f872cd05080 1 bucket sync caught up with source: 2021-02-11T09:53:16.259 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000004.44.9, , ] 2021-02-11T09:53:16.259 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000004.44.9, , ] 2021-02-11T09:53:16.260 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:16.259+0000 7f872cd05080 0 bucket checkpoint complete 2021-02-11T09:53:16.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:16.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:16.305 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:16.348 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:16.349 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:16.564 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:16.563+0000 7f73407a6080 1 bucket sync caught up with source: 2021-02-11T09:53:16.564 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000004.44.9, , ] 2021-02-11T09:53:16.565 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000004.44.9, , ] 2021-02-11T09:53:16.565 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:16.563+0000 7f73407a6080 0 bucket checkpoint complete 2021-02-11T09:53:16.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:16.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:16.597 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:16.686 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:16.687 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:16.911+0000 7f006261b080 1 bucket sync caught up with source: 2021-02-11T09:53:16.913 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000004.29.9, , ] 2021-02-11T09:53:16.913 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000004.29.9, , ] 2021-02-11T09:53:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:16.911+0000 7f006261b080 0 bucket checkpoint complete 2021-02-11T09:53:16.941 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:16.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:16.942 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:16.990 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:16.990 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:17.200 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:17.197+0000 7fcea2b0c080 1 bucket sync caught up with source: 2021-02-11T09:53:17.200 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000004.29.9, , ] 2021-02-11T09:53:17.200 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000004.29.9, , ] 2021-02-11T09:53:17.200 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:17.197+0000 7fcea2b0c080 0 bucket checkpoint complete 2021-02-11T09:53:17.234 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-20 zones={test-zone2, test-zone1} 2021-02-11T09:53:18.010 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:18.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:18.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:18.018 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:18.066 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:18.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:18.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:18.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:18.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:18.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:18.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:18.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:18.414 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-20 zones={test-zone2, test-zone1} 2021-02-11T09:53:18.415 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-20 zones={test-zone3, test-zone1} 2021-02-11T09:53:19.093 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:19.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:19.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:19.139 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:19.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:19.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:19.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:19.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:19.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:19.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:19.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:19.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:19.479 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-20 zones={test-zone3, test-zone1} 2021-02-11T09:53:19.479 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-20 zones={test-zone3, test-zone2} 2021-02-11T09:53:19.609 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:19.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:19.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:19.618 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:19.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:19.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:19.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:19.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:19.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:19.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:19.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:19.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:19.999 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-20 zones={test-zone3, test-zone2} 2021-02-11T09:53:20.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:20.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:20.015 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:20.052 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:20.052 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:20.235 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f2716509-104d-423b-ba5c-db1bd74642ab 2021-02-11T09:53:20.275 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613036985.070217_101.1","last_update":"2021-02-11T09:49:45.070217Z"},{"marker":"1_1613037200.012526_214.1","last_update":"2021-02-11T09:53:20.012526Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:53:20.275 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:53:20.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:20.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:20.275 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:20.316 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:20.316 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:20.547 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037200.012526_214.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:20.012526Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:20.547 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:53:20.547 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:53:20.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:20.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:20.548 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:20.588 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:20.588 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:20.828 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037200.012526_214.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:20.012526Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:20.828 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:53:20.829 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:53:20.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:20.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:20.829 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:20.870 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:20.871 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:21.087 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037200.012526_214.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:20.012526Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:21.088 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:53:22.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:22.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:22.047 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:22.094 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:22.094 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:22.297 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:22.293+0000 7ff9307cd080 1 bucket sync caught up with source: 2021-02-11T09:53:22.297 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000005.24.17, , ] 2021-02-11T09:53:22.298 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000005.24.17, , ] 2021-02-11T09:53:22.298 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:22.293+0000 7ff9307cd080 0 bucket checkpoint complete 2021-02-11T09:53:22.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:22.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:22.332 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:22.382 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:22.382 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:22.595 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:22.591+0000 7fda82526080 1 bucket sync caught up with source: 2021-02-11T09:53:22.595 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000005.24.17, , ] 2021-02-11T09:53:22.595 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000005.24.17, , ] 2021-02-11T09:53:22.595 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:22.591+0000 7fda82526080 0 bucket checkpoint complete 2021-02-11T09:53:22.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:22.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:22.623 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:22.671 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:22.671 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:22.879 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:22.877+0000 7f6a226b7080 1 bucket sync caught up with source: 2021-02-11T09:53:22.880 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000005.24.17, , ] 2021-02-11T09:53:22.880 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000005.24.17, , ] 2021-02-11T09:53:22.880 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:22.877+0000 7f6a226b7080 0 bucket checkpoint complete 2021-02-11T09:53:22.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:22.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:22.914 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:22.962 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:22.962 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:23.180 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:23.179+0000 7ff195c73080 1 bucket sync caught up with source: 2021-02-11T09:53:23.180 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000007.52.14, , ] 2021-02-11T09:53:23.180 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000007.52.14, , ] 2021-02-11T09:53:23.180 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:23.179+0000 7ff195c73080 0 bucket checkpoint complete 2021-02-11T09:53:23.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:23.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:23.217 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:23.264 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:23.264 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:23.504 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:23.503+0000 7f191deaf080 1 bucket sync caught up with source: 2021-02-11T09:53:23.504 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000007.52.14, , ] 2021-02-11T09:53:23.504 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000007.52.14, , ] 2021-02-11T09:53:23.504 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:23.503+0000 7f191deaf080 0 bucket checkpoint complete 2021-02-11T09:53:23.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:23.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:23.539 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:23.583 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:23.583 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:23.820 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:23.819+0000 7f67cfd59080 1 bucket sync caught up with source: 2021-02-11T09:53:23.820 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000007.37.14, , ] 2021-02-11T09:53:23.820 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000007.37.14, , ] 2021-02-11T09:53:23.820 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:23.819+0000 7f67cfd59080 0 bucket checkpoint complete 2021-02-11T09:53:23.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:23.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:23.848 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:23.896 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:23.896 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:24.119 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:24.113+0000 7f3eb8d02080 1 bucket sync caught up with source: 2021-02-11T09:53:24.119 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000007.37.14, , ] 2021-02-11T09:53:24.119 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000007.37.14, , ] 2021-02-11T09:53:24.119 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:24.113+0000 7f3eb8d02080 0 bucket checkpoint complete 2021-02-11T09:53:24.152 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-20 zones={test-zone2, test-zone1} 2021-02-11T09:53:25.951 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:25.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:25.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:25.960 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:26.011 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:26.011 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:26.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:26.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:26.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:26.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:26.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:26.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:26.343 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-20 zones={test-zone2, test-zone1} 2021-02-11T09:53:26.343 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-20 zones={test-zone3, test-zone1} 2021-02-11T09:53:26.446 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:26.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:26.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:26.495 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:26.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:26.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:26.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:26.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:26.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:26.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:26.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:26.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:26.861 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-20 zones={test-zone3, test-zone1} 2021-02-11T09:53:26.861 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-20 zones={test-zone3, test-zone2} 2021-02-11T09:53:26.871 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:26.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:26.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:26.919 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:26.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:26.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:26.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:26.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:27.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:27.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:27.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:27.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:27.261 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-20 zones={test-zone3, test-zone2} 2021-02-11T09:53:27.261 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2021-02-11T09:53:27.262 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-21 2021-02-11T09:53:27.300 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-22 2021-02-11T09:53:27.378 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-23 2021-02-11T09:53:27.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:27.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:27.510 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:27.547 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:27.547 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:27.742 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f2716509-104d-423b-ba5c-db1bd74642ab 2021-02-11T09:53:27.778 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613036985.070217_101.1","last_update":"2021-02-11T09:49:45.070217Z"},{"marker":"1_1613037207.496661_232.1","last_update":"2021-02-11T09:53:27.496661Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:53:27.778 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:53:27.778 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:27.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:27.779 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:27.819 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:27.819 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:28.051 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037207.496661_232.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:27.496661Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:28.051 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:53:28.051 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:53:28.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:28.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:28.052 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:28.094 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:28.094 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:28.345 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037207.496661_232.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:27.496661Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:28.345 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:53:28.345 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:53:28.345 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:28.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:28.346 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:28.385 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:28.385 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:28.613 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037207.496661_232.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:27.496661Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:28.614 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:53:31.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:31.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:31.133 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:31.180 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:31.180 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:31.390 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:31.385+0000 7f1691e2d080 1 bucket sync caught up with source: 2021-02-11T09:53:31.390 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000003.13.11, , ] 2021-02-11T09:53:31.390 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000003.13.11, , ] 2021-02-11T09:53:31.390 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:31.385+0000 7f1691e2d080 0 bucket checkpoint complete 2021-02-11T09:53:31.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:31.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:31.428 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:31.480 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:31.480 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:31.698 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:31.695+0000 7fafff4ca080 1 bucket sync caught up with source: 2021-02-11T09:53:31.698 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000003.13.11, , ] 2021-02-11T09:53:31.698 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000003.13.11, , ] 2021-02-11T09:53:31.698 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:31.695+0000 7fafff4ca080 0 bucket checkpoint complete 2021-02-11T09:53:31.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:31.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:31.727 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:31.775 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:31.775 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:31.983 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:31.976+0000 7fd84fabe080 1 bucket sync caught up with source: 2021-02-11T09:53:31.983 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000003.13.11, , ] 2021-02-11T09:53:31.983 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000003.13.11, , ] 2021-02-11T09:53:31.983 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:31.976+0000 7fd84fabe080 0 bucket checkpoint complete 2021-02-11T09:53:32.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:32.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:32.016 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:32.065 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:32.065 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:32.296 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:32.295+0000 7f27a532b080 1 bucket sync caught up with source: 2021-02-11T09:53:32.296 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000002.16.5, , ] 2021-02-11T09:53:32.296 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000002.16.5, , ] 2021-02-11T09:53:32.296 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:32.295+0000 7f27a532b080 0 bucket checkpoint complete 2021-02-11T09:53:32.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:32.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:32.331 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:32.375 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:32.375 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:32.594 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:32.591+0000 7f442d28a080 1 bucket sync caught up with source: 2021-02-11T09:53:32.594 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000002.16.5, , ] 2021-02-11T09:53:32.594 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000002.16.5, , ] 2021-02-11T09:53:32.594 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:32.591+0000 7f442d28a080 0 bucket checkpoint complete 2021-02-11T09:53:32.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:32.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:32.630 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:32.675 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:32.676 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:32.879 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:32.879+0000 7fed58ac0080 1 bucket sync caught up with source: 2021-02-11T09:53:32.879 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000002.21.5, , ] 2021-02-11T09:53:32.880 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000002.21.5, , ] 2021-02-11T09:53:32.880 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:32.879+0000 7fed58ac0080 0 bucket checkpoint complete 2021-02-11T09:53:32.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:32.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:32.912 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:32.960 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:32.960 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:33.183 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:33.176+0000 7fd1d821e080 1 bucket sync caught up with source: 2021-02-11T09:53:33.183 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000002.21.5, , ] 2021-02-11T09:53:33.184 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000002.21.5, , ] 2021-02-11T09:53:33.184 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:33.176+0000 7fd1d821e080 0 bucket checkpoint complete 2021-02-11T09:53:33.219 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-21 zones={test-zone2, test-zone1} 2021-02-11T09:53:34.689 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:34.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:34.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:34.696 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:34.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:34.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:34.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:34.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:34.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:34.889 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-21 zones={test-zone2, test-zone1} 2021-02-11T09:53:34.889 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-21 zones={test-zone3, test-zone1} 2021-02-11T09:53:35.625 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:35.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:35.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:35.671 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:35.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:35.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:35.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:35.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:35.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:35.821 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-21 zones={test-zone3, test-zone1} 2021-02-11T09:53:35.821 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-21 zones={test-zone3, test-zone2} 2021-02-11T09:53:35.832 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:35.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:35.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:35.879 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:35.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:35.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:35.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:35.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:35.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:36.030 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-21 zones={test-zone3, test-zone2} 2021-02-11T09:53:36.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:36.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:36.031 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:36.078 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:36.078 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:36.289 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:36.284+0000 7f402c9c9080 1 bucket sync caught up with source: 2021-02-11T09:53:36.289 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000002.9.5, , ] 2021-02-11T09:53:36.289 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000002.9.5, , ] 2021-02-11T09:53:36.289 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:36.284+0000 7f402c9c9080 0 bucket checkpoint complete 2021-02-11T09:53:36.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:36.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:36.324 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:36.373 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:36.373 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:36.593 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:36.591+0000 7f63d4f8c080 1 bucket sync caught up with source: 2021-02-11T09:53:36.593 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000002.9.5, , ] 2021-02-11T09:53:36.593 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000002.9.5, , ] 2021-02-11T09:53:36.593 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:36.591+0000 7f63d4f8c080 0 bucket checkpoint complete 2021-02-11T09:53:36.625 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:36.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:36.626 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:36.673 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:36.673 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:36.882 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:36.876+0000 7fe690523080 1 bucket sync caught up with source: 2021-02-11T09:53:36.882 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000002.9.5, , ] 2021-02-11T09:53:36.882 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000002.9.5, , ] 2021-02-11T09:53:36.882 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:36.876+0000 7fe690523080 0 bucket checkpoint complete 2021-02-11T09:53:36.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:36.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:36.912 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:36.960 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:36.961 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:37.205 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:37.203+0000 7f0b7eccb080 1 bucket sync caught up with source: 2021-02-11T09:53:37.205 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000003.15.11, , ] 2021-02-11T09:53:37.205 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000003.15.11, , ] 2021-02-11T09:53:37.205 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:37.203+0000 7f0b7eccb080 0 bucket checkpoint complete 2021-02-11T09:53:37.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:37.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:37.239 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:37.285 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:37.285 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:37.506 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:37.503+0000 7fa7eacc3080 1 bucket sync caught up with source: 2021-02-11T09:53:37.507 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000003.15.11, , ] 2021-02-11T09:53:37.507 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000003.15.11, , ] 2021-02-11T09:53:37.507 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:37.503+0000 7fa7eacc3080 0 bucket checkpoint complete 2021-02-11T09:53:37.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:37.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:37.544 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:37.590 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:37.590 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:37.802 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:37.799+0000 7f3505827080 1 bucket sync caught up with source: 2021-02-11T09:53:37.802 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000002.14.5, , ] 2021-02-11T09:53:37.802 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000002.14.5, , ] 2021-02-11T09:53:37.802 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:37.799+0000 7f3505827080 0 bucket checkpoint complete 2021-02-11T09:53:37.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:37.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:37.835 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:37.884 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:37.884 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:38.112 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:38.108+0000 7f19cca03080 1 bucket sync caught up with source: 2021-02-11T09:53:38.113 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000002.14.5, , ] 2021-02-11T09:53:38.113 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000002.14.5, , ] 2021-02-11T09:53:38.113 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:38.108+0000 7f19cca03080 0 bucket checkpoint complete 2021-02-11T09:53:38.146 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-22 zones={test-zone2, test-zone1} 2021-02-11T09:53:38.913 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:38.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:38.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:38.920 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:38.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:38.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:39.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:39.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:39.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:39.117 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-22 zones={test-zone2, test-zone1} 2021-02-11T09:53:39.117 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-22 zones={test-zone3, test-zone1} 2021-02-11T09:53:39.381 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:39.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:39.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:39.427 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:39.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:39.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:39.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:39.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:39.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:39.568 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-22 zones={test-zone3, test-zone1} 2021-02-11T09:53:39.568 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-22 zones={test-zone3, test-zone2} 2021-02-11T09:53:39.578 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:39.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:39.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:39.631 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:39.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:39.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:39.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:39.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:39.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:39.781 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-22 zones={test-zone3, test-zone2} 2021-02-11T09:53:39.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:39.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:39.781 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:39.826 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:39.827 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:40.061 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:40.056+0000 7fa73f0fd080 1 bucket sync caught up with source: 2021-02-11T09:53:40.061 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000002.12.5, , ] 2021-02-11T09:53:40.061 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000002.12.5, , ] 2021-02-11T09:53:40.061 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:40.056+0000 7fa73f0fd080 0 bucket checkpoint complete 2021-02-11T09:53:40.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:40.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:40.096 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:40.146 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:40.146 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:40.371 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:40.370+0000 7f12e0d60080 1 bucket sync caught up with source: 2021-02-11T09:53:40.371 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000002.12.5, , ] 2021-02-11T09:53:40.371 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000002.12.5, , ] 2021-02-11T09:53:40.371 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:40.370+0000 7f12e0d60080 0 bucket checkpoint complete 2021-02-11T09:53:40.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:40.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:40.404 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:40.451 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:40.451 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:40.673 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:40.668+0000 7f56b5f9a080 1 bucket sync caught up with source: 2021-02-11T09:53:40.674 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000002.12.5, , ] 2021-02-11T09:53:40.674 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000002.12.5, , ] 2021-02-11T09:53:40.674 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:40.668+0000 7f56b5f9a080 0 bucket checkpoint complete 2021-02-11T09:53:40.706 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:40.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:40.706 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:40.753 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:40.753 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:40.983 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:40.978+0000 7f162fe36080 1 bucket sync caught up with source: 2021-02-11T09:53:40.983 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000002.21.5, , ] 2021-02-11T09:53:40.983 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000002.21.5, , ] 2021-02-11T09:53:40.983 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:40.978+0000 7f162fe36080 0 bucket checkpoint complete 2021-02-11T09:53:41.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:41.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:41.018 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:41.289 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:41.286+0000 7f576edfb080 1 bucket sync caught up with source: 2021-02-11T09:53:41.290 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000002.21.5, , ] 2021-02-11T09:53:41.290 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000002.21.5, , ] 2021-02-11T09:53:41.290 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:41.286+0000 7f576edfb080 0 bucket checkpoint complete 2021-02-11T09:53:41.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:41.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:41.322 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:41.368 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:41.368 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:41.592 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:41.590+0000 7fb57fb26080 1 bucket sync caught up with source: 2021-02-11T09:53:41.592 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000003.21.11, , ] 2021-02-11T09:53:41.592 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000003.21.11, , ] 2021-02-11T09:53:41.592 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:41.590+0000 7fb57fb26080 0 bucket checkpoint complete 2021-02-11T09:53:41.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:41.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:41.621 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:41.667 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:41.667 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:41.891 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:41.884+0000 7f4279aac080 1 bucket sync caught up with source: 2021-02-11T09:53:41.891 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000003.21.11, , ] 2021-02-11T09:53:41.891 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000003.21.11, , ] 2021-02-11T09:53:41.891 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:41.884+0000 7f4279aac080 0 bucket checkpoint complete 2021-02-11T09:53:41.924 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-23 zones={test-zone2, test-zone1} 2021-02-11T09:53:43.331 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:43.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:43.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:43.338 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:43.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:43.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:43.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:43.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:43.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:43.529 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-23 zones={test-zone2, test-zone1} 2021-02-11T09:53:43.529 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-23 zones={test-zone3, test-zone1} 2021-02-11T09:53:44.523 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:44.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:44.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:44.571 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:44.576 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:44.576 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:44.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:44.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:44.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:44.716 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-23 zones={test-zone3, test-zone1} 2021-02-11T09:53:44.716 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-23 zones={test-zone3, test-zone2} 2021-02-11T09:53:44.964 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:44.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:44.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:44.971 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:45.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:45.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:45.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:45.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:45.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:45.161 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-23 zones={test-zone3, test-zone2} 2021-02-11T09:53:45.161 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2021-02-11T09:53:45.162 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-24 2021-02-11T09:53:45.201 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-25 2021-02-11T09:53:45.267 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-26 2021-02-11T09:53:45.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:45.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:45.450 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:45.486 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:45.486 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:45.701 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f2716509-104d-423b-ba5c-db1bd74642ab 2021-02-11T09:53:45.739 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613036985.070217_101.1","last_update":"2021-02-11T09:49:45.070217Z"},{"marker":"1_1613037225.441045_256.1","last_update":"2021-02-11T09:53:45.441045Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:53:45.740 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:53:45.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:45.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:45.740 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:45.776 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:45.776 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:46.006 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037225.441045_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:45.441045Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:46.006 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:53:46.006 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:53:46.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:46.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:46.007 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:46.048 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:46.048 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:46.287 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037225.441045_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:45.441045Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:46.288 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:53:46.288 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:53:46.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:46.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:46.288 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:46.328 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:46.329 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:46.567 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037225.441045_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:45.441045Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:46.567 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:53:48.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:48.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:48.131 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:48.179 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:48.179 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:48.408 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:48.404+0000 7f9affe58080 1 bucket sync caught up with source: 2021-02-11T09:53:48.408 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000003.45.6, , ] 2021-02-11T09:53:48.409 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000003.45.6, , ] 2021-02-11T09:53:48.409 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:48.404+0000 7f9affe58080 0 bucket checkpoint complete 2021-02-11T09:53:48.442 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:48.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:48.443 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:48.491 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:48.491 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:48.697 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:48.694+0000 7f30d4c6e080 1 bucket sync caught up with source: 2021-02-11T09:53:48.697 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000003.45.6, , ] 2021-02-11T09:53:48.697 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000003.45.6, , ] 2021-02-11T09:53:48.697 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:48.694+0000 7f30d4c6e080 0 bucket checkpoint complete 2021-02-11T09:53:48.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:48.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:48.727 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:48.772 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:48.773 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:48.969 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:48.964+0000 7f30efd69080 1 bucket sync caught up with source: 2021-02-11T09:53:48.969 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000003.45.6, , ] 2021-02-11T09:53:48.969 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000003.45.6, , ] 2021-02-11T09:53:48.969 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:48.964+0000 7f30efd69080 0 bucket checkpoint complete 2021-02-11T09:53:49.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:49.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:49.000 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:49.050 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:49.050 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:49.276 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:49.274+0000 7fc87a066080 1 bucket sync caught up with source: 2021-02-11T09:53:49.277 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000001.67.12, , ] 2021-02-11T09:53:49.277 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000001.67.12, , ] 2021-02-11T09:53:49.277 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:49.274+0000 7fc87a066080 0 bucket checkpoint complete 2021-02-11T09:53:49.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:49.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:49.314 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:49.361 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:49.361 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:49.595 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:49.590+0000 7f2136807080 1 bucket sync caught up with source: 2021-02-11T09:53:49.595 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000001.67.12, , ] 2021-02-11T09:53:49.595 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000001.67.12, , ] 2021-02-11T09:53:49.595 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:49.590+0000 7f2136807080 0 bucket checkpoint complete 2021-02-11T09:53:49.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:49.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:49.629 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:49.673 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:49.674 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:49.884 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:49.882+0000 7fbb88254080 1 bucket sync caught up with source: 2021-02-11T09:53:49.884 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000001.70.12, , ] 2021-02-11T09:53:49.885 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000001.70.12, , ] 2021-02-11T09:53:49.885 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:49.882+0000 7fbb88254080 0 bucket checkpoint complete 2021-02-11T09:53:49.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:49.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:49.921 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:49.968 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:49.968 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:50.195 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:50.188+0000 7f5ea4bcd080 1 bucket sync caught up with source: 2021-02-11T09:53:50.195 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000001.70.12, , ] 2021-02-11T09:53:50.195 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000001.70.12, , ] 2021-02-11T09:53:50.195 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:50.188+0000 7f5ea4bcd080 0 bucket checkpoint complete 2021-02-11T09:53:50.228 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-24 zones={test-zone2, test-zone1} 2021-02-11T09:53:51.859 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:51.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:51.867 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:51.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:51.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:51.975 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-24 zones={test-zone2, test-zone1} 2021-02-11T09:53:51.976 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-24 zones={test-zone3, test-zone1} 2021-02-11T09:53:52.980 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:53.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:53.027 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:53.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:53.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:53.094 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-24 zones={test-zone3, test-zone1} 2021-02-11T09:53:53.094 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-24 zones={test-zone3, test-zone2} 2021-02-11T09:53:53.759 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:53.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:53.767 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:53.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:53.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:53.876 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-24 zones={test-zone3, test-zone2} 2021-02-11T09:53:53.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:53.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:53.877 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:53.924 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:53.924 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:54.153 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:54.148+0000 7fb375c0d080 1 bucket sync caught up with source: 2021-02-11T09:53:54.153 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000001.28.12, , ] 2021-02-11T09:53:54.153 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000001.28.12, , ] 2021-02-11T09:53:54.153 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:54.148+0000 7fb375c0d080 0 bucket checkpoint complete 2021-02-11T09:53:54.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:54.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:54.187 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:54.235 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:54.235 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:54.478 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:54.474+0000 7f4e5163e080 1 bucket sync caught up with source: 2021-02-11T09:53:54.478 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000001.28.12, , ] 2021-02-11T09:53:54.478 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000001.28.12, , ] 2021-02-11T09:53:54.478 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:54.474+0000 7f4e5163e080 0 bucket checkpoint complete 2021-02-11T09:53:54.509 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:54.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:54.510 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:54.557 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:54.557 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:54.778 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:54.772+0000 7f9be2b14080 1 bucket sync caught up with source: 2021-02-11T09:53:54.778 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000001.28.12, , ] 2021-02-11T09:53:54.778 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000001.28.12, , ] 2021-02-11T09:53:54.778 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:54.772+0000 7f9be2b14080 0 bucket checkpoint complete 2021-02-11T09:53:54.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:54.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:54.812 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:54.860 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:55.093 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:55.090+0000 7f5d777d0080 1 bucket sync caught up with source: 2021-02-11T09:53:55.093 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000003.65.6, , ] 2021-02-11T09:53:55.093 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000003.65.6, , ] 2021-02-11T09:53:55.094 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:55.090+0000 7f5d777d0080 0 bucket checkpoint complete 2021-02-11T09:53:55.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:55.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:55.129 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:55.396 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:55.394+0000 7fd82f367080 1 bucket sync caught up with source: 2021-02-11T09:53:55.396 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000003.65.6, , ] 2021-02-11T09:53:55.396 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000003.65.6, , ] 2021-02-11T09:53:55.396 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:55.394+0000 7fd82f367080 0 bucket checkpoint complete 2021-02-11T09:53:55.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:55.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:55.429 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:55.471 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:55.471 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:55.679 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:55.678+0000 7fb2de684080 1 bucket sync caught up with source: 2021-02-11T09:53:55.680 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000001.41.12, , ] 2021-02-11T09:53:55.680 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000001.41.12, , ] 2021-02-11T09:53:55.680 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:55.678+0000 7fb2de684080 0 bucket checkpoint complete 2021-02-11T09:53:55.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:55.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:55.710 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:55.756 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:55.756 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:55.965 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:55.960+0000 7fe1108c4080 1 bucket sync caught up with source: 2021-02-11T09:53:55.966 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000001.41.12, , ] 2021-02-11T09:53:55.966 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000001.41.12, , ] 2021-02-11T09:53:55.966 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:55.960+0000 7fe1108c4080 0 bucket checkpoint complete 2021-02-11T09:53:55.998 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-25 zones={test-zone2, test-zone1} 2021-02-11T09:53:56.858 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:56.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:56.866 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:56.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:56.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:56.976 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-25 zones={test-zone2, test-zone1} 2021-02-11T09:53:56.976 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-25 zones={test-zone3, test-zone1} 2021-02-11T09:53:57.172 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:57.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:57.219 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:57.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:57.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:57.287 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-25 zones={test-zone3, test-zone1} 2021-02-11T09:53:57.287 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-25 zones={test-zone3, test-zone2} 2021-02-11T09:53:57.298 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:57.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:57.347 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:57.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:53:57.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:53:57.415 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-25 zones={test-zone3, test-zone2} 2021-02-11T09:53:57.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:57.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:57.415 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:57.460 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:57.460 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:57.682 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:57.676+0000 7fcd52288080 1 bucket sync caught up with source: 2021-02-11T09:53:57.682 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000001.16.12, , ] 2021-02-11T09:53:57.682 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000001.16.12, , ] 2021-02-11T09:53:57.682 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:57.676+0000 7fcd52288080 0 bucket checkpoint complete 2021-02-11T09:53:57.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:57.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:57.716 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:57.767 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:57.767 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:57.995 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:57.994+0000 7f0791a06080 1 bucket sync caught up with source: 2021-02-11T09:53:57.996 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000001.16.12, , ] 2021-02-11T09:53:57.996 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000001.16.12, , ] 2021-02-11T09:53:57.996 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:57.994+0000 7f0791a06080 0 bucket checkpoint complete 2021-02-11T09:53:58.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:58.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:58.028 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:58.077 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:58.077 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:58.289 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:58.284+0000 7fbb060d3080 1 bucket sync caught up with source: 2021-02-11T09:53:58.289 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000001.16.12, , ] 2021-02-11T09:53:58.289 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000001.16.12, , ] 2021-02-11T09:53:58.290 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:58.284+0000 7fbb060d3080 0 bucket checkpoint complete 2021-02-11T09:53:58.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:58.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:58.321 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:58.371 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:58.371 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:58.603 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:58.598+0000 7fc8f966f080 1 bucket sync caught up with source: 2021-02-11T09:53:58.603 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000001.37.12, , ] 2021-02-11T09:53:58.603 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000001.37.12, , ] 2021-02-11T09:53:58.603 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:58.598+0000 7fc8f966f080 0 bucket checkpoint complete 2021-02-11T09:53:58.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:58.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:58.638 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:58.683 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:58.683 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:58.902 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:58.898+0000 7f192dcd2080 1 bucket sync caught up with source: 2021-02-11T09:53:58.902 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000001.37.12, , ] 2021-02-11T09:53:58.902 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000001.37.12, , ] 2021-02-11T09:53:58.902 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:58.898+0000 7f192dcd2080 0 bucket checkpoint complete 2021-02-11T09:53:58.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:58.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:58.937 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:58.984 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:58.984 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:59.208 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:59.206+0000 7f4cbe049080 1 bucket sync caught up with source: 2021-02-11T09:53:59.208 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , 00000000003.36.6, , ] 2021-02-11T09:53:59.208 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , 00000000003.36.6, , ] 2021-02-11T09:53:59.208 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:53:59.206+0000 7f4cbe049080 0 bucket checkpoint complete 2021-02-11T09:53:59.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:59.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:53:59.243 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:53:59.290 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:59.290 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:59.514 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:59.508+0000 7f476971f080 1 bucket sync caught up with source: 2021-02-11T09:53:59.515 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , 00000000003.36.6, , ] 2021-02-11T09:53:59.515 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , 00000000003.36.6, , ] 2021-02-11T09:53:59.515 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:53:59.508+0000 7f476971f080 0 bucket checkpoint complete 2021-02-11T09:53:59.546 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-26 zones={test-zone2, test-zone1} 2021-02-11T09:54:00.795 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:54:00.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:54:00.802 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:54:00.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:54:00.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:54:00.912 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-26 zones={test-zone2, test-zone1} 2021-02-11T09:54:00.912 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-26 zones={test-zone3, test-zone1} 2021-02-11T09:54:01.615 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:54:01.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:54:01.663 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:54:01.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:54:01.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:54:01.726 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-26 zones={test-zone3, test-zone1} 2021-02-11T09:54:01.726 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-26 zones={test-zone3, test-zone2} 2021-02-11T09:54:01.736 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:54:01.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:54:01.782 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:54:01.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:54:01.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:54:02.643 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-26 zones={test-zone3, test-zone2} 2021-02-11T09:54:02.644 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2021-02-11T09:54:02.644 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-27 2021-02-11T09:54:02.678 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-28 2021-02-11T09:54:02.737 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-29 2021-02-11T09:54:03.002 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2021-02-11T09:54:03.003 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-30 2021-02-11T09:54:03.037 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-31 2021-02-11T09:54:03.092 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-32 2021-02-11T09:54:03.351 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2021-02-11T09:54:03.351 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-33 2021-02-11T09:54:03.399 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-34 2021-02-11T09:54:03.452 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-35 2021-02-11T09:54:06.223 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2021-02-11T09:54:06.223 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-36 2021-02-11T09:54:06.724 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-37 2021-02-11T09:54:07.359 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-38 2021-02-11T09:54:08.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:08.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:08.335 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:54:08.377 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:54:08.377 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:54:08.569 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f2716509-104d-423b-ba5c-db1bd74642ab 2021-02-11T09:54:08.608 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613036985.070217_101.1","last_update":"2021-02-11T09:49:45.070217Z"},{"marker":"1_1613037248.306636_324.1","last_update":"2021-02-11T09:54:08.306636Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:54:08.608 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:54:08.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:08.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:08.608 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:54:08.647 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:54:08.647 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:54:08.886 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037248.306636_324.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:54:08.306636Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:54:08.887 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:54:08.887 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:54:08.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:08.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:08.887 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:54:08.929 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:54:08.930 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:54:09.179 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037248.306636_324.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:54:08.306636Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:54:09.179 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:54:09.179 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:54:09.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:09.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:09.180 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:54:09.221 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:54:09.221 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:54:09.465 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037248.306636_324.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:54:08.306636Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:54:09.466 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:54:12.570 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2021-02-11T09:54:12.571 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-39 2021-02-11T09:54:12.605 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-40 2021-02-11T09:54:13.004 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-41 2021-02-11T09:54:13.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:13.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:13.360 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:54:13.395 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:54:13.395 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:54:13.579 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f2716509-104d-423b-ba5c-db1bd74642ab 2021-02-11T09:54:13.623 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613036985.070217_101.1","last_update":"2021-02-11T09:49:45.070217Z"},{"marker":"1_1613037253.337510_336.1","last_update":"2021-02-11T09:54:13.337510Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:54:13.623 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:54:13.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:13.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:13.624 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:54:13.665 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:54:13.665 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:54:13.902 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037253.337510_336.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:54:13.337510Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:54:13.903 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:54:13.903 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:54:13.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:13.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:13.903 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:54:13.949 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:54:13.949 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:54:14.206 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037253.337510_336.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:54:13.337510Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:54:14.206 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:54:14.206 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:54:14.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:14.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:14.207 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:54:14.250 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:54:14.250 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:54:14.479 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f2716509-104d-423b-ba5c-db1bd74642ab","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037253.337510_336.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:54:13.337510Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:54:14.479 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:54:14.479 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-02-11T09:54:14.479 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T09:54:14.480 INFO:tasks.rgw.c1.client.1.smithi026.stdout:2021-02-11T09:54:14.478+0000 7f6912348700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 19922) UID: 0 2021-02-11T09:54:14.480 INFO:tasks.rgw.c1.client.1.smithi026.stdout:2021-02-11T09:54:14.478+0000 7f6929ea30c0 -1 shutting down 2021-02-11T09:54:18.718 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-02-11T09:54:20.581 INFO:tasks.rgw.c1.client.1:Stopped 2021-02-11T09:54:20.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:54:20.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:54:20.582 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:54:20.620 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:54:20.620 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:54:20.645 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:54:20.640+0000 7f4a170b5080 0 NOTICE: overriding master zone: a3c4910a-4529-4294-aed4-ef409f209cf5 2021-02-11T09:54:20.646 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:54:20.640+0000 7f4a170b5080 0 failed reading obj info from .rgw.root:zone_info.30c2afa9-98cd-42de-acef-f4a612d65cea: (2) No such file or directory 2021-02-11T09:54:20.646 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:54:20.640+0000 7f4a170b5080 0 WARNING: could not read zone params for zone id=30c2afa9-98cd-42de-acef-f4a612d65cea name=test-zone3 2021-02-11T09:54:20.647 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:54:20.644+0000 7f4a170b5080 0 failed reading obj info from .rgw.root:zone_info.a3c4910a-4529-4294-aed4-ef409f209cf5: (2) No such file or directory 2021-02-11T09:54:20.647 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:54:20.644+0000 7f4a170b5080 0 WARNING: could not read zone params for zone id=a3c4910a-4529-4294-aed4-ef409f209cf5 name=test-zone1 2021-02-11T09:54:20.661 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"DHA6OWMQ42PGF1T7","secret_key":"xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T09:54:20.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:54:20.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:54:20.661 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:54:20.740 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:54:20.741 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:54:21.059 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"8ead83cf-fd6c-4023-bc3d-a277ca9366d6","epoch":1,"predecessor_uuid":"f2716509-104d-423b-ba5c-db1bd74642ab","sync_status":["1_1613036985.070217_101.1","1_1613037253.337510_336.1","",""],"period_map":{"id":"8ead83cf-fd6c-4023-bc3d-a277ca9366d6","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","zones":[{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"30c2afa9-98cd-42de-acef-f4a612d65cea","name":"test-zone3","endpoints":["http://smithi026.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"a3c4910a-4529-4294-aed4-ef409f209cf5","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","val":3299024572},{"key":"30c2afa9-98cd-42de-acef-f4a612d65cea","val":2754623789},{"key":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","val":3684997273},{"key":"a3c4910a-4529-4294-aed4-ef409f209cf5","val":2693415762}]},"master_zonegroup":"test-zonegroup","master_zone":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","realm_name":"test-realm","realm_epoch":3} 2021-02-11T09:54:21.060 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2021-02-11T09:54:24.322 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-02-11T09:55:21.061 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-42 2021-02-11T09:55:21.134 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-43 2021-02-11T09:55:21.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:21.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:21.173 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:55:21.210 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:55:21.210 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:55:21.421 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=8ead83cf-fd6c-4023-bc3d-a277ca9366d6 2021-02-11T09:55:21.458 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613037321.168858_350.1","last_update":"2021-02-11T09:55:21.168858Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:55:21.459 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2021-02-11T09:55:21.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:21.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:21.459 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:55:21.502 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:55:21.502 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:55:21.748 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8ead83cf-fd6c-4023-bc3d-a277ca9366d6","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":80,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"1_1613037321.168858_350.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:55:21.168858Z","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":80,"complete":80}} 2021-02-11T09:55:21.748 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2021-02-11T09:55:21.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:55:21.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:55:21.749 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:55:21.825 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:55:21.826 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:55:21.852 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:55:21.849+0000 7f2574d4f080 0 NOTICE: overriding master zone: 2f5f3613-ec60-4b48-a757-91ddd4a3ec48 2021-02-11T09:55:21.852 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:55:21.849+0000 7f2574d4f080 0 failed reading obj info from .rgw.root:zone_info.2f5f3613-ec60-4b48-a757-91ddd4a3ec48: (2) No such file or directory 2021-02-11T09:55:21.853 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:55:21.849+0000 7f2574d4f080 0 WARNING: could not read zone params for zone id=2f5f3613-ec60-4b48-a757-91ddd4a3ec48 name=test-zone2 2021-02-11T09:55:21.854 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:55:21.849+0000 7f2574d4f080 0 failed reading obj info from .rgw.root:zone_info.57f4d9d8-b2db-4f2f-b58c-d908c07c1a88: (2) No such file or directory 2021-02-11T09:55:21.854 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T09:55:21.849+0000 7f2574d4f080 0 WARNING: could not read zone params for zone id=57f4d9d8-b2db-4f2f-b58c-d908c07c1a88 name=test-zone4 2021-02-11T09:55:21.873 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"a3c4910a-4529-4294-aed4-ef409f209cf5","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"DHA6OWMQ42PGF1T7","secret_key":"xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","notif_pool":"test-zone1.rgw.log:notif"} 2021-02-11T09:55:21.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:55:21.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:55:21.874 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:55:21.955 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:55:21.955 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:55:22.266 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","epoch":1,"predecessor_uuid":"8ead83cf-fd6c-4023-bc3d-a277ca9366d6","sync_status":["","1_1613037321.168858_350.1","",""],"period_map":{"id":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","zones":[{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"30c2afa9-98cd-42de-acef-f4a612d65cea","name":"test-zone3","endpoints":["http://smithi026.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"a3c4910a-4529-4294-aed4-ef409f209cf5","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","val":3299024572},{"key":"30c2afa9-98cd-42de-acef-f4a612d65cea","val":2754623789},{"key":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","val":3684997273},{"key":"a3c4910a-4529-4294-aed4-ef409f209cf5","val":2693415762}]},"master_zonegroup":"test-zonegroup","master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","realm_name":"test-realm","realm_epoch":4} 2021-02-11T09:55:22.266 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2021-02-11T09:56:22.267 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-44 2021-02-11T09:56:22.314 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-45 2021-02-11T09:56:22.389 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-02-11T09:56:22.389 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-02-11T09:56:22.392 INFO:tasks.rgw.c1.client.1:Started 2021-02-11T09:56:22.392 DEBUG:teuthology.orchestra.run.smithi026:> curl http://smithi026.front.sepia.ceph.com:8001 2021-02-11T09:56:22.406 INFO:teuthology.orchestra.run.smithi026.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:56:22.407 INFO:teuthology.orchestra.run.smithi026.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:56:22.411 INFO:teuthology.orchestra.run.smithi026.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi026.front.sepia.ceph.com port 8001: Connection refused 2021-02-11T09:56:22.412 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T09:56:23.413 DEBUG:teuthology.orchestra.run.smithi026:> curl http://smithi026.front.sepia.ceph.com:8001 2021-02-11T09:56:23.429 INFO:teuthology.orchestra.run.smithi026.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:56:23.429 INFO:teuthology.orchestra.run.smithi026.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:56:23.436 INFO:teuthology.orchestra.run.smithi026.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2021-02-11T09:56:23.437 INFO:teuthology.orchestra.run.smithi026.stdout:anonymous 2021-02-11T09:56:23.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:23.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:23.437 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:23.522 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:23.522 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:23.723 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T09:56:23.758 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613037382.362758_155.1","last_update":"2021-02-11T09:56:22.362758Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:56:23.758 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:56:23.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:23.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:23.759 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:23.801 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:23.801 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:24.033 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037382.362758_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:22.362758Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:56:24.033 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:56:24.033 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:56:24.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:24.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:24.034 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:24.075 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:24.075 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:24.327 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037382.362758_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:22.362758Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:56:24.327 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:56:24.327 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:56:24.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:24.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:24.327 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:24.369 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:24.369 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:24.602 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037382.362758_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:22.362758Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:56:24.603 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:56:24.603 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-39 zones={test-zone2, test-zone1} 2021-02-11T09:56:26.530 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:26.537 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:26.638 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-39 zones={test-zone2, test-zone1} 2021-02-11T09:56:26.638 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-39 zones={test-zone3, test-zone1} 2021-02-11T09:56:26.660 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:26.710 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:26.767 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-39 zones={test-zone3, test-zone1} 2021-02-11T09:56:26.768 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-39 zones={test-zone1, test-zone2} 2021-02-11T09:56:26.777 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:26.822 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:26.881 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-39 zones={test-zone1, test-zone2} 2021-02-11T09:56:26.882 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-39 zones={test-zone3, test-zone2} 2021-02-11T09:56:26.890 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:26.938 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:27.002 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-39 zones={test-zone3, test-zone2} 2021-02-11T09:56:27.002 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-39 zones={test-zone1, test-zone3} 2021-02-11T09:56:27.013 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:27.062 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:27.125 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-39 zones={test-zone1, test-zone3} 2021-02-11T09:56:27.125 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-39 zones={test-zone2, test-zone3} 2021-02-11T09:56:27.136 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:27.182 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:27.245 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-39 zones={test-zone2, test-zone3} 2021-02-11T09:56:27.245 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-40 zones={test-zone2, test-zone1} 2021-02-11T09:56:27.260 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:27.307 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:27.368 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-40 zones={test-zone2, test-zone1} 2021-02-11T09:56:27.368 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-40 zones={test-zone3, test-zone1} 2021-02-11T09:56:27.380 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:27.426 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:27.490 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-40 zones={test-zone3, test-zone1} 2021-02-11T09:56:27.490 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-40 zones={test-zone1, test-zone2} 2021-02-11T09:56:27.499 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:27.546 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:27.608 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-40 zones={test-zone1, test-zone2} 2021-02-11T09:56:27.608 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-40 zones={test-zone3, test-zone2} 2021-02-11T09:56:27.619 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:27.666 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:27.729 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-40 zones={test-zone3, test-zone2} 2021-02-11T09:56:27.729 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-40 zones={test-zone1, test-zone3} 2021-02-11T09:56:27.742 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:27.790 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:27.848 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-40 zones={test-zone1, test-zone3} 2021-02-11T09:56:27.848 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-40 zones={test-zone2, test-zone3} 2021-02-11T09:56:27.858 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:27.906 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:27.969 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-40 zones={test-zone2, test-zone3} 2021-02-11T09:56:27.969 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-41 zones={test-zone2, test-zone1} 2021-02-11T09:56:27.985 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:28.034 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:28.096 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-41 zones={test-zone2, test-zone1} 2021-02-11T09:56:28.097 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-41 zones={test-zone3, test-zone1} 2021-02-11T09:56:28.108 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:28.154 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:28.212 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-41 zones={test-zone3, test-zone1} 2021-02-11T09:56:28.212 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-41 zones={test-zone1, test-zone2} 2021-02-11T09:56:28.222 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:28.270 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:28.336 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-41 zones={test-zone1, test-zone2} 2021-02-11T09:56:28.336 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-41 zones={test-zone3, test-zone2} 2021-02-11T09:56:28.346 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:28.394 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:28.452 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-41 zones={test-zone3, test-zone2} 2021-02-11T09:56:28.452 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-41 zones={test-zone1, test-zone3} 2021-02-11T09:56:28.462 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:28.510 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:28.572 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-41 zones={test-zone1, test-zone3} 2021-02-11T09:56:28.572 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-41 zones={test-zone2, test-zone3} 2021-02-11T09:56:28.579 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:28.626 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:28.685 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-41 zones={test-zone2, test-zone3} 2021-02-11T09:56:28.685 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-42 zones={test-zone2, test-zone1} 2021-02-11T09:56:28.694 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:28.742 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:28.804 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-42 zones={test-zone2, test-zone1} 2021-02-11T09:56:28.804 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-42 zones={test-zone3, test-zone1} 2021-02-11T09:56:28.813 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:28.858 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:28.916 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-42 zones={test-zone3, test-zone1} 2021-02-11T09:56:28.916 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-42 zones={test-zone1, test-zone2} 2021-02-11T09:56:28.927 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:28.974 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:29.033 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-42 zones={test-zone1, test-zone2} 2021-02-11T09:56:29.033 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-42 zones={test-zone3, test-zone2} 2021-02-11T09:56:29.043 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:29.090 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:29.157 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-42 zones={test-zone3, test-zone2} 2021-02-11T09:56:29.157 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-42 zones={test-zone1, test-zone3} 2021-02-11T09:56:29.168 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:29.214 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:29.273 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-42 zones={test-zone1, test-zone3} 2021-02-11T09:56:29.273 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-42 zones={test-zone2, test-zone3} 2021-02-11T09:56:29.282 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:29.330 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:29.393 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-42 zones={test-zone2, test-zone3} 2021-02-11T09:56:29.393 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-43 zones={test-zone2, test-zone1} 2021-02-11T09:56:29.402 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:29.455 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:29.518 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-43 zones={test-zone2, test-zone1} 2021-02-11T09:56:29.518 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-43 zones={test-zone3, test-zone1} 2021-02-11T09:56:29.528 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:29.574 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:29.641 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-43 zones={test-zone3, test-zone1} 2021-02-11T09:56:29.641 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-43 zones={test-zone1, test-zone2} 2021-02-11T09:56:29.651 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:29.698 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:29.766 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-43 zones={test-zone1, test-zone2} 2021-02-11T09:56:29.766 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-43 zones={test-zone3, test-zone2} 2021-02-11T09:56:29.778 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:29.831 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:29.898 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-43 zones={test-zone3, test-zone2} 2021-02-11T09:56:29.898 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-43 zones={test-zone1, test-zone3} 2021-02-11T09:56:29.910 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:29.962 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:30.025 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-43 zones={test-zone1, test-zone3} 2021-02-11T09:56:30.025 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-43 zones={test-zone2, test-zone3} 2021-02-11T09:56:30.036 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:30.082 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:30.146 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-43 zones={test-zone2, test-zone3} 2021-02-11T09:56:30.146 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-44 zones={test-zone2, test-zone1} 2021-02-11T09:56:30.156 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:30.202 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:30.261 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-44 zones={test-zone2, test-zone1} 2021-02-11T09:56:30.261 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-44 zones={test-zone3, test-zone1} 2021-02-11T09:56:30.269 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:30.314 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:30.376 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-44 zones={test-zone3, test-zone1} 2021-02-11T09:56:30.376 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-44 zones={test-zone1, test-zone2} 2021-02-11T09:56:30.386 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:30.430 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:30.492 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-44 zones={test-zone1, test-zone2} 2021-02-11T09:56:30.492 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-44 zones={test-zone3, test-zone2} 2021-02-11T09:56:30.502 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:30.550 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:30.612 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-44 zones={test-zone3, test-zone2} 2021-02-11T09:56:30.612 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-44 zones={test-zone1, test-zone3} 2021-02-11T09:56:30.623 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:30.670 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:30.733 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-44 zones={test-zone1, test-zone3} 2021-02-11T09:56:30.733 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-44 zones={test-zone2, test-zone3} 2021-02-11T09:56:30.743 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:30.790 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:30.848 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-44 zones={test-zone2, test-zone3} 2021-02-11T09:56:30.848 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-45 zones={test-zone2, test-zone1} 2021-02-11T09:56:30.857 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:30.906 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:30.970 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-45 zones={test-zone2, test-zone1} 2021-02-11T09:56:30.970 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-45 zones={test-zone3, test-zone1} 2021-02-11T09:56:30.980 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:31.026 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:31.088 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-45 zones={test-zone3, test-zone1} 2021-02-11T09:56:31.088 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-45 zones={test-zone1, test-zone2} 2021-02-11T09:56:31.098 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:31.142 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:31.206 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-45 zones={test-zone1, test-zone2} 2021-02-11T09:56:31.206 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-45 zones={test-zone3, test-zone2} 2021-02-11T09:56:31.215 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:31.262 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:31.321 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-45 zones={test-zone3, test-zone2} 2021-02-11T09:56:31.321 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-45 zones={test-zone1, test-zone3} 2021-02-11T09:56:31.333 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:31.382 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:31.444 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-45 zones={test-zone1, test-zone3} 2021-02-11T09:56:31.445 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-45 zones={test-zone2, test-zone3} 2021-02-11T09:56:31.454 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:56:31.503 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:56:31.565 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-45 zones={test-zone2, test-zone3} 2021-02-11T09:56:31.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f2716509-104d-423b-ba5c-db1bd74642ab', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:31.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f2716509-104d-423b-ba5c-db1bd74642ab', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:31.566 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f2716509-104d-423b-ba5c-db1bd74642ab --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:31.603 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:31.603 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:31.871 INFO:teuthology.orchestra.run.smithi026.stdout:[{"id":"1_1613036845.801758_43.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T09:47:25.801758607Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lbhm83ChfGn6EHKoaQzp5TB","ver":1},"status":{"status":"write"}}},{"id":"1_1613036845.807024_48.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T09:47:25.807024959Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lbhm83ChfGn6EHKoaQzp5TB","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036964.781099_55.1","section":"bucket.instance","name":"erivkc-1:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.1","timestamp":"2021-02-11T09:49:24.781099770Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7CX9vbVJ-GdTYgNGNLhgoE1","ver":1},"status":{"status":"write"}}},{"id":"1_1613036964.794012_57.1","section":"bucket.instance","name":"erivkc-1:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.1","timestamp":"2021-02-11T09:49:24.794012217Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7CX9vbVJ-GdTYgNGNLhgoE1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036964.798399_58.1","section":"bucket","name":"erivkc-1","timestamp":"2021-02-11T09:49:24.798399196Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CPPkcG3v2weIBmWYso1zTaV","ver":1},"status":{"status":"write"}}},{"id":"1_1613036964.804170_59.1","section":"bucket","name":"erivkc-1","timestamp":"2021-02-11T09:49:24.804170325Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CPPkcG3v2weIBmWYso1zTaV","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036964.833958_60.1","section":"bucket.instance","name":"erivkc-2:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.2","timestamp":"2021-02-11T09:49:24.833958584Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2FGD8auz1ZhEdsT6v2xSbGp","ver":1},"status":{"status":"write"}}},{"id":"1_1613036964.841848_61.1","section":"bucket.instance","name":"erivkc-2:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.2","timestamp":"2021-02-11T09:49:24.841848223Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2FGD8auz1ZhEdsT6v2xSbGp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036964.843748_62.1","section":"bucket","name":"erivkc-2","timestamp":"2021-02-11T09:49:24.843748711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N0rMsI_g5Yis-zEde5_K0Ir","ver":1},"status":{"status":"write"}}},{"id":"1_1613036964.850242_63.1","section":"bucket","name":"erivkc-2","timestamp":"2021-02-11T09:49:24.850242744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N0rMsI_g5Yis-zEde5_K0Ir","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036968.651442_64.1","section":"bucket.instance","name":"erivkc-3:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.3","timestamp":"2021-02-11T09:49:28.651442746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1S35tybN40BRSEaKI2l3DIk","ver":1},"status":{"status":"write"}}},{"id":"1_1613036968.660175_65.1","section":"bucket.instance","name":"erivkc-3:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.3","timestamp":"2021-02-11T09:49:28.660175151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1S35tybN40BRSEaKI2l3DIk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036968.662383_66.1","section":"bucket","name":"erivkc-3","timestamp":"2021-02-11T09:49:28.662383146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aUbSv47O20-wQgZ_f_BQyws","ver":1},"status":{"status":"write"}}},{"id":"1_1613036968.672641_67.1","section":"bucket","name":"erivkc-3","timestamp":"2021-02-11T09:49:28.672641364Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aUbSv47O20-wQgZ_f_BQyws","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036972.560421_68.1","section":"bucket.instance","name":"erivkc-4:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.4","timestamp":"2021-02-11T09:49:32.560421693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PBJ_0UMaWtCXBaplvwFqFoX","ver":1},"status":{"status":"write"}}},{"id":"1_1613036972.565575_70.1","section":"bucket.instance","name":"erivkc-4:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.4","timestamp":"2021-02-11T09:49:32.565575234Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PBJ_0UMaWtCXBaplvwFqFoX","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036972.567393_71.1","section":"bucket","name":"erivkc-4","timestamp":"2021-02-11T09:49:32.567393622Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1FZcpPqSVgDrFiCSVeHkB-p","ver":1},"status":{"status":"write"}}},{"id":"1_1613036972.572741_72.1","section":"bucket","name":"erivkc-4","timestamp":"2021-02-11T09:49:32.572741163Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1FZcpPqSVgDrFiCSVeHkB-p","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036972.892360_73.1","section":"bucket.instance","name":"erivkc-5:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.5","timestamp":"2021-02-11T09:49:32.892360275Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_97AcbvI_qrWCvEAT4gZODL3","ver":1},"status":{"status":"write"}}},{"id":"1_1613036972.897238_74.1","section":"bucket.instance","name":"erivkc-5:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.5","timestamp":"2021-02-11T09:49:32.897238243Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_97AcbvI_qrWCvEAT4gZODL3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036972.899057_75.1","section":"bucket","name":"erivkc-5","timestamp":"2021-02-11T09:49:32.899057233Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j8qWxRQVbGg7UOusTEwf7B9","ver":1},"status":{"status":"write"}}},{"id":"1_1613036972.904781_76.1","section":"bucket","name":"erivkc-5","timestamp":"2021-02-11T09:49:32.904781579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j8qWxRQVbGg7UOusTEwf7B9","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036973.578464_77.1","section":"bucket.instance","name":"erivkc-6:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.6","timestamp":"2021-02-11T09:49:33.578464448Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y3I0bk2WnnKmeCxS7P_RG4F","ver":1},"status":{"status":"write"}}},{"id":"1_1613036973.583582_78.1","section":"bucket.instance","name":"erivkc-6:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.6","timestamp":"2021-02-11T09:49:33.583582832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y3I0bk2WnnKmeCxS7P_RG4F","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036973.585257_79.1","section":"bucket","name":"erivkc-6","timestamp":"2021-02-11T09:49:33.585257871Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a53snF3YGjGzzmA--K1O94L","ver":1},"status":{"status":"write"}}},{"id":"1_1613036973.589655_80.1","section":"bucket","name":"erivkc-6","timestamp":"2021-02-11T09:49:33.589655678Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a53snF3YGjGzzmA--K1O94L","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.632140_81.1","section":"bucket.instance","name":"erivkc-7:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.7","timestamp":"2021-02-11T09:49:41.632140668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KTbG7jYEk3iEpbVAS1YRyaR","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.637460_84.1","section":"bucket.instance","name":"erivkc-7:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.7","timestamp":"2021-02-11T09:49:41.637460231Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KTbG7jYEk3iEpbVAS1YRyaR","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.640349_85.1","section":"bucket","name":"erivkc-7","timestamp":"2021-02-11T09:49:41.640349109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.647069_86.1","section":"bucket","name":"erivkc-7","timestamp":"2021-02-11T09:49:41.647069461Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.667580_87.1","section":"bucket.instance","name":"erivkc-8:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.8","timestamp":"2021-02-11T09:49:41.667580530Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dg-o8rJKnYjuar4EHq-FKMN","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.675956_88.1","section":"bucket.instance","name":"erivkc-8:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.8","timestamp":"2021-02-11T09:49:41.675956821Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dg-o8rJKnYjuar4EHq-FKMN","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.678555_89.1","section":"bucket","name":"erivkc-8","timestamp":"2021-02-11T09:49:41.678555790Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.684839_90.1","section":"bucket","name":"erivkc-8","timestamp":"2021-02-11T09:49:41.684839859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.745183_91.1","section":"bucket.instance","name":"erivkc-9:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.9","timestamp":"2021-02-11T09:49:41.745183419Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W5J5xaWTI0Gmgb3HWRJ8MbG","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.749950_92.1","section":"bucket.instance","name":"erivkc-9:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.9","timestamp":"2021-02-11T09:49:41.749950640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W5J5xaWTI0Gmgb3HWRJ8MbG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.751718_93.1","section":"bucket","name":"erivkc-9","timestamp":"2021-02-11T09:49:41.751718057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.757312_94.1","section":"bucket","name":"erivkc-9","timestamp":"2021-02-11T09:49:41.757312185Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036984.846663_95.1","section":"bucket","name":"erivkc-7","timestamp":"2021-02-11T09:49:44.846663475Z","data":{"read_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":1},"write_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036984.853017_97.1","section":"bucket","name":"erivkc-7","timestamp":"2021-02-11T09:49:44.853017326Z","data":{"read_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":1},"write_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036984.929328_98.1","section":"bucket","name":"erivkc-8","timestamp":"2021-02-11T09:49:44.929328641Z","data":{"read_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":1},"write_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036984.936781_99.1","section":"bucket","name":"erivkc-8","timestamp":"2021-02-11T09:49:44.936781341Z","data":{"read_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":1},"write_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036985.064812_100.1","section":"bucket","name":"erivkc-9","timestamp":"2021-02-11T09:49:45.064812325Z","data":{"read_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":1},"write_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036985.070217_101.1","section":"bucket","name":"erivkc-9","timestamp":"2021-02-11T09:49:45.070217328Z","data":{"read_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":1},"write_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036999.845976_50.1","section":"bucket.instance","name":"erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10","timestamp":"2021-02-11T09:49:59.845976867Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X61poZNvtY7wornc-IkI9Ue","ver":1},"status":{"status":"write"}}},{"id":"1_1613036999.851424_52.1","section":"bucket.instance","name":"erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10","timestamp":"2021-02-11T09:49:59.851424369Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X61poZNvtY7wornc-IkI9Ue","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036999.853338_53.1","section":"bucket","name":"erivkc-10","timestamp":"2021-02-11T09:49:59.853338236Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f9x9v20LyN2Z56R02tSPZg7","ver":1},"status":{"status":"write"}}},{"id":"1_1613036999.858076_54.1","section":"bucket","name":"erivkc-10","timestamp":"2021-02-11T09:49:59.858076108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f9x9v20LyN2Z56R02tSPZg7","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036999.993526_56.1","section":"bucket.instance","name":"erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11","timestamp":"2021-02-11T09:49:59.993526854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c9r9KX8B4gD8WQmfwj_PhZZ","ver":1},"status":{"status":"write"}}},{"id":"1_1613036999.999923_57.1","section":"bucket.instance","name":"erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11","timestamp":"2021-02-11T09:49:59.999923814Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c9r9KX8B4gD8WQmfwj_PhZZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037000.001777_58.1","section":"bucket","name":"erivkc-11","timestamp":"2021-02-11T09:50:00.001777802Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dqr2cFllRHl99RGY2i9FuEy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037000.007381_59.1","section":"bucket","name":"erivkc-11","timestamp":"2021-02-11T09:50:00.007381054Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dqr2cFllRHl99RGY2i9FuEy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037000.051806_60.1","section":"bucket.instance","name":"erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12","timestamp":"2021-02-11T09:50:00.051806024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D8OCKLAmYRpy1715HaEnUrg","ver":1},"status":{"status":"write"}}},{"id":"1_1613037000.057515_61.1","section":"bucket.instance","name":"erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12","timestamp":"2021-02-11T09:50:00.057515941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D8OCKLAmYRpy1715HaEnUrg","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037000.061137_62.1","section":"bucket","name":"erivkc-12","timestamp":"2021-02-11T09:50:00.061137438Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AkdgcL9F7rK58axzIzhY-5R","ver":1},"status":{"status":"write"}}},{"id":"1_1613037000.074230_63.1","section":"bucket","name":"erivkc-12","timestamp":"2021-02-11T09:50:00.074230334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AkdgcL9F7rK58axzIzhY-5R","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.704912_167.1","section":"bucket.instance","name":"erivkc-13:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.13","timestamp":"2021-02-11T09:50:13.704912580Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wtf4rO_4_UFnmTOV6Ka-w-5","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.710270_168.1","section":"bucket.instance","name":"erivkc-13:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.13","timestamp":"2021-02-11T09:50:13.710270887Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wtf4rO_4_UFnmTOV6Ka-w-5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.712243_169.1","section":"bucket","name":"erivkc-13","timestamp":"2021-02-11T09:50:13.712243833Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xIInLmSguo5ojG9rkSrh6NV","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.717978_170.1","section":"bucket","name":"erivkc-13","timestamp":"2021-02-11T09:50:13.717978080Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xIInLmSguo5ojG9rkSrh6NV","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.741489_171.1","section":"bucket.instance","name":"erivkc-14:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.14","timestamp":"2021-02-11T09:50:13.741489385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qjqHLxJu0bVGtSl_DLxSzVc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.748782_172.1","section":"bucket.instance","name":"erivkc-14:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.14","timestamp":"2021-02-11T09:50:13.748782975Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qjqHLxJu0bVGtSl_DLxSzVc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.750682_173.1","section":"bucket","name":"erivkc-14","timestamp":"2021-02-11T09:50:13.750682310Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w3Y_zBpyOfitYjQtUFfUTuw","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.755278_174.1","section":"bucket","name":"erivkc-14","timestamp":"2021-02-11T09:50:13.755278974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w3Y_zBpyOfitYjQtUFfUTuw","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.798885_175.1","section":"bucket.instance","name":"erivkc-15:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.15","timestamp":"2021-02-11T09:50:13.798885547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8XLPKECf7E8u3tos2GxeuTy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.804452_176.1","section":"bucket.instance","name":"erivkc-15:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.15","timestamp":"2021-02-11T09:50:13.804452901Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8XLPKECf7E8u3tos2GxeuTy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.806359_177.1","section":"bucket","name":"erivkc-15","timestamp":"2021-02-11T09:50:13.806359983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9Prt9QKeINJuoGyCp0FIuyH","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.812144_178.1","section":"bucket","name":"erivkc-15","timestamp":"2021-02-11T09:50:13.812144543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9Prt9QKeINJuoGyCp0FIuyH","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037041.978668_183.1","section":"bucket.instance","name":"erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16","timestamp":"2021-02-11T09:50:41.978668726Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":1},"status":{"status":"write"}}},{"id":"1_1613037041.984135_184.1","section":"bucket.instance","name":"erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16","timestamp":"2021-02-11T09:50:41.984135248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037041.986045_185.1","section":"bucket","name":"erivkc-16","timestamp":"2021-02-11T09:50:41.986045547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RZKX2QLjfnqRLbyJ9ygdT-6","ver":1},"status":{"status":"write"}}},{"id":"1_1613037041.991151_186.1","section":"bucket","name":"erivkc-16","timestamp":"2021-02-11T09:50:41.991151568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RZKX2QLjfnqRLbyJ9ygdT-6","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037042.691423_187.1","section":"bucket.instance","name":"erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17","timestamp":"2021-02-11T09:50:42.691423169Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":1},"status":{"status":"write"}}},{"id":"1_1613037042.696754_188.1","section":"bucket.instance","name":"erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17","timestamp":"2021-02-11T09:50:42.696754644Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037042.698816_189.1","section":"bucket","name":"erivkc-17","timestamp":"2021-02-11T09:50:42.698816857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z7VmmfI938ifTeJ9aE0V1Ml","ver":1},"status":{"status":"write"}}},{"id":"1_1613037042.703975_190.1","section":"bucket","name":"erivkc-17","timestamp":"2021-02-11T09:50:42.703975143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z7VmmfI938ifTeJ9aE0V1Ml","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037043.653339_191.1","section":"bucket.instance","name":"erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18","timestamp":"2021-02-11T09:50:43.653339978Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":1},"status":{"status":"write"}}},{"id":"1_1613037043.659257_192.1","section":"bucket.instance","name":"erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18","timestamp":"2021-02-11T09:50:43.659257779Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037043.661142_193.1","section":"bucket","name":"erivkc-18","timestamp":"2021-02-11T09:50:43.661142206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GjOszKvzPokzjfI2YAb1ciu","ver":1},"status":{"status":"write"}}},{"id":"1_1613037043.666578_194.1","section":"bucket","name":"erivkc-18","timestamp":"2021-02-11T09:50:43.666578075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GjOszKvzPokzjfI2YAb1ciu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037044.503349_195.1","section":"bucket.instance","name":"erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16","timestamp":"2021-02-11T09:50:44.503349589Z","data":{"read_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":1},"write_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":2},"status":{"status":"write"}}},{"id":"1_1613037044.514489_196.1","section":"bucket.instance","name":"erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16","timestamp":"2021-02-11T09:50:44.514489077Z","data":{"read_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":1},"write_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037045.153847_197.1","section":"bucket.instance","name":"erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17","timestamp":"2021-02-11T09:50:45.153847525Z","data":{"read_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":1},"write_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":2},"status":{"status":"write"}}},{"id":"1_1613037045.165645_198.1","section":"bucket.instance","name":"erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17","timestamp":"2021-02-11T09:50:45.165645371Z","data":{"read_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":1},"write_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037045.642925_199.1","section":"bucket.instance","name":"erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18","timestamp":"2021-02-11T09:50:45.642925401Z","data":{"read_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":1},"write_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":2},"status":{"status":"write"}}},{"id":"1_1613037045.651083_200.1","section":"bucket.instance","name":"erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18","timestamp":"2021-02-11T09:50:45.651083022Z","data":{"read_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":1},"write_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037125.481929_201.1","section":"bucket.instance","name":"erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19","timestamp":"2021-02-11T09:52:05.481929899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.487401_202.1","section":"bucket.instance","name":"erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19","timestamp":"2021-02-11T09:52:05.487401571Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.489373_203.1","section":"bucket","name":"erivkc-19","timestamp":"2021-02-11T09:52:05.489373157Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Jff8SGfEPMevwcwi_i-z2vJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.498307_204.1","section":"bucket","name":"erivkc-19","timestamp":"2021-02-11T09:52:05.498307854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Jff8SGfEPMevwcwi_i-z2vJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.505367_205.1","section":"bucket.instance","name":"erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19","timestamp":"2021-02-11T09:52:05.505367585Z","data":{"read_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":1},"write_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":2},"status":{"status":"write"}}},{"id":"1_1613037125.514387_206.1","section":"bucket.instance","name":"erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19","timestamp":"2021-02-11T09:52:05.514387451Z","data":{"read_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":1},"write_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037186.354909_207.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:06.354909614Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.360229_208.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:06.360229344Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037186.362241_209.1","section":"bucket","name":"erivkc-20","timestamp":"2021-02-11T09:53:06.362241066Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f2qv734GBzTTVNMfY72MNwy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.368386_210.1","section":"bucket","name":"erivkc-20","timestamp":"2021-02-11T09:53:06.368386631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f2qv734GBzTTVNMfY72MNwy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037193.931707_211.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:13.931707918Z","data":{"read_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":1},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":2},"status":{"status":"write"}}},{"id":"1_1613037193.942339_212.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:13.942339208Z","data":{"read_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":1},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037200.002098_213.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:20.002098607Z","data":{"read_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":2},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":3},"status":{"status":"write"}}},{"id":"1_1613037200.012526_214.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:20.012526463Z","data":{"read_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":2},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037207.282029_215.1","section":"bucket.instance","name":"erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21","timestamp":"2021-02-11T09:53:27.282029137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.288219_216.1","section":"bucket.instance","name":"erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21","timestamp":"2021-02-11T09:53:27.288219184Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.290439_217.1","section":"bucket","name":"erivkc-21","timestamp":"2021-02-11T09:53:27.290439853Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tvXoIwFs9HpMEOc0iK5QT1h","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.296030_218.1","section":"bucket","name":"erivkc-21","timestamp":"2021-02-11T09:53:27.296030630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tvXoIwFs9HpMEOc0iK5QT1h","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.320765_219.1","section":"bucket.instance","name":"erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22","timestamp":"2021-02-11T09:53:27.320765881Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.326634_220.1","section":"bucket.instance","name":"erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22","timestamp":"2021-02-11T09:53:27.326634616Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.328915_221.1","section":"bucket","name":"erivkc-22","timestamp":"2021-02-11T09:53:27.328915473Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zb993XEJ8BiM8mBe3hvWtO0","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.334920_222.1","section":"bucket","name":"erivkc-22","timestamp":"2021-02-11T09:53:27.334920903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zb993XEJ8BiM8mBe3hvWtO0","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.394575_223.1","section":"bucket.instance","name":"erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23","timestamp":"2021-02-11T09:53:27.394575034Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.403410_224.1","section":"bucket.instance","name":"erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23","timestamp":"2021-02-11T09:53:27.403410972Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.408075_225.1","section":"bucket","name":"erivkc-23","timestamp":"2021-02-11T09:53:27.408075932Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x_zF6jY24d41dOixQ9zJGrE","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.414692_226.1","section":"bucket","name":"erivkc-23","timestamp":"2021-02-11T09:53:27.414692449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x_zF6jY24d41dOixQ9zJGrE","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.445616_227.1","section":"bucket.instance","name":"erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21","timestamp":"2021-02-11T09:53:27.445616320Z","data":{"read_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":1},"write_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":2},"status":{"status":"write"}}},{"id":"1_1613037207.454205_228.1","section":"bucket.instance","name":"erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21","timestamp":"2021-02-11T09:53:27.454205301Z","data":{"read_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":1},"write_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037207.463414_229.1","section":"bucket.instance","name":"erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22","timestamp":"2021-02-11T09:53:27.463414545Z","data":{"read_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":1},"write_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":2},"status":{"status":"write"}}},{"id":"1_1613037207.472168_230.1","section":"bucket.instance","name":"erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22","timestamp":"2021-02-11T09:53:27.472168737Z","data":{"read_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":1},"write_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037207.485112_231.1","section":"bucket.instance","name":"erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23","timestamp":"2021-02-11T09:53:27.485112100Z","data":{"read_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":1},"write_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":2},"status":{"status":"write"}}},{"id":"1_1613037207.496661_232.1","section":"bucket.instance","name":"erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23","timestamp":"2021-02-11T09:53:27.496661758Z","data":{"read_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":1},"write_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037225.179784_233.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.179784590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.187474_234.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.187474262Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.189621_235.1","section":"bucket","name":"erivkc-24","timestamp":"2021-02-11T09:53:45.189621882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KKbtq-SaZp1BbP3NTY0W1Qw","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.196093_236.1","section":"bucket","name":"erivkc-24","timestamp":"2021-02-11T09:53:45.196093466Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KKbtq-SaZp1BbP3NTY0W1Qw","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.222139_237.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.222139168Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.230640_238.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.230640700Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.232853_239.1","section":"bucket","name":"erivkc-25","timestamp":"2021-02-11T09:53:45.232853969Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v3PBdFYuFlkcRuNZcqiRP0s","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.239328_240.1","section":"bucket","name":"erivkc-25","timestamp":"2021-02-11T09:53:45.239328358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v3PBdFYuFlkcRuNZcqiRP0s","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.288768_241.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.288768050Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.296414_242.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.296414547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.298349_243.1","section":"bucket","name":"erivkc-26","timestamp":"2021-02-11T09:53:45.298349270Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3HLmRlz5MeuRgXHTkAYCO5-","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.307308_244.1","section":"bucket","name":"erivkc-26","timestamp":"2021-02-11T09:53:45.307308476Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3HLmRlz5MeuRgXHTkAYCO5-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.338581_245.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.338581131Z","data":{"read_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":1},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":2},"status":{"status":"write"}}},{"id":"1_1613037225.345749_246.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.345749522Z","data":{"read_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":1},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037225.350555_247.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.350555471Z","data":{"read_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":2},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":3},"status":{"status":"write"}}},{"id":"1_1613037225.361606_248.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.361606024Z","data":{"read_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":2},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037225.372379_249.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.372379079Z","data":{"read_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":1},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":2},"status":{"status":"write"}}},{"id":"1_1613037225.386914_250.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.386914935Z","data":{"read_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":1},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037225.397898_251.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.397898539Z","data":{"read_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":2},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":3},"status":{"status":"write"}}},{"id":"1_1613037225.403904_252.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.403904049Z","data":{"read_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":2},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037225.419818_253.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.419818566Z","data":{"read_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":1},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":2},"status":{"status":"write"}}},{"id":"1_1613037225.424682_254.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.424682186Z","data":{"read_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":1},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037225.434889_255.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.434889809Z","data":{"read_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":2},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":3},"status":{"status":"write"}}},{"id":"1_1613037225.441045_256.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.441045732Z","data":{"read_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":2},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037242.662969_257.1","section":"bucket.instance","name":"erivkc-27:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.27","timestamp":"2021-02-11T09:54:02.662969410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.668739_258.1","section":"bucket.instance","name":"erivkc-27:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.27","timestamp":"2021-02-11T09:54:02.668739755Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.670752_259.1","section":"bucket","name":"erivkc-27","timestamp":"2021-02-11T09:54:02.670752769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gH_oydOqixBGu2q9mVl-QS_","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.675376_260.1","section":"bucket","name":"erivkc-27","timestamp":"2021-02-11T09:54:02.675376669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gH_oydOqixBGu2q9mVl-QS_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.700651_261.1","section":"bucket.instance","name":"erivkc-28:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.28","timestamp":"2021-02-11T09:54:02.700651797Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.706443_262.1","section":"bucket.instance","name":"erivkc-28:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.28","timestamp":"2021-02-11T09:54:02.706443830Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.708331_263.1","section":"bucket","name":"erivkc-28","timestamp":"2021-02-11T09:54:02.708331319Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTuUd6ahhMShO2l9OhA78Xc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.713018_264.1","section":"bucket","name":"erivkc-28","timestamp":"2021-02-11T09:54:02.713018965Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTuUd6ahhMShO2l9OhA78Xc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.762579_265.1","section":"bucket.instance","name":"erivkc-29:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.29","timestamp":"2021-02-11T09:54:02.762579200Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.775474_266.1","section":"bucket.instance","name":"erivkc-29:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.29","timestamp":"2021-02-11T09:54:02.775474031Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.782163_267.1","section":"bucket","name":"erivkc-29","timestamp":"2021-02-11T09:54:02.782163490Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OcZh0JgP4psaTeCxR4eUFzB","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.789419_268.1","section":"bucket","name":"erivkc-29","timestamp":"2021-02-11T09:54:02.789419763Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OcZh0JgP4psaTeCxR4eUFzB","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.824967_270.1","section":"bucket.instance","name":"erivkc-27:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.27","timestamp":"2021-02-11T09:54:02.824967755Z","data":{"read_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":1},"write_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":2},"status":{"status":"write"}}},{"id":"1_1613037242.831294_271.1","section":"bucket.instance","name":"erivkc-27:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.27","timestamp":"2021-02-11T09:54:02.831294730Z","data":{"read_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":1},"write_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037242.879129_272.1","section":"bucket.instance","name":"erivkc-28:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.28","timestamp":"2021-02-11T09:54:02.879129288Z","data":{"read_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":1},"write_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":2},"status":{"status":"write"}}},{"id":"1_1613037242.888573_273.1","section":"bucket.instance","name":"erivkc-28:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.28","timestamp":"2021-02-11T09:54:02.888573312Z","data":{"read_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":1},"write_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037242.941315_274.1","section":"bucket.instance","name":"erivkc-29:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.29","timestamp":"2021-02-11T09:54:02.941315096Z","data":{"read_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":1},"write_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":2},"status":{"status":"write"}}},{"id":"1_1613037242.953261_275.1","section":"bucket.instance","name":"erivkc-29:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.29","timestamp":"2021-02-11T09:54:02.953261205Z","data":{"read_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":1},"write_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037243.018597_277.1","section":"bucket.instance","name":"erivkc-30:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.30","timestamp":"2021-02-11T09:54:03.018597532Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.024482_278.1","section":"bucket.instance","name":"erivkc-30:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.30","timestamp":"2021-02-11T09:54:03.024482726Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.026590_279.1","section":"bucket","name":"erivkc-30","timestamp":"2021-02-11T09:54:03.026590985Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b8xacbbSicSxMqmYhFg2pvb","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.032758_280.1","section":"bucket","name":"erivkc-30","timestamp":"2021-02-11T09:54:03.032758243Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b8xacbbSicSxMqmYhFg2pvb","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.060231_281.1","section":"bucket.instance","name":"erivkc-31:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.31","timestamp":"2021-02-11T09:54:03.060231918Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.066319_282.1","section":"bucket.instance","name":"erivkc-31:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.31","timestamp":"2021-02-11T09:54:03.066319997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.068298_283.1","section":"bucket","name":"erivkc-31","timestamp":"2021-02-11T09:54:03.068298381Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DRK-e9sS3ydFf_VBlB5b3Uc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.072941_284.1","section":"bucket","name":"erivkc-31","timestamp":"2021-02-11T09:54:03.072941983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DRK-e9sS3ydFf_VBlB5b3Uc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.112419_285.1","section":"bucket.instance","name":"erivkc-32:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.32","timestamp":"2021-02-11T09:54:03.112419548Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.119240_286.1","section":"bucket.instance","name":"erivkc-32:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.32","timestamp":"2021-02-11T09:54:03.119240251Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.121798_287.1","section":"bucket","name":"erivkc-32","timestamp":"2021-02-11T09:54:03.121798440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_axnp7YglcFxPJLhOhhHHGZS","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.128266_288.1","section":"bucket","name":"erivkc-32","timestamp":"2021-02-11T09:54:03.128266708Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_axnp7YglcFxPJLhOhhHHGZS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.157131_289.1","section":"bucket.instance","name":"erivkc-30:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.30","timestamp":"2021-02-11T09:54:03.157131853Z","data":{"read_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":1},"write_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":2},"status":{"status":"write"}}},{"id":"1_1613037243.168904_290.1","section":"bucket.instance","name":"erivkc-30:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.30","timestamp":"2021-02-11T09:54:03.168904231Z","data":{"read_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":1},"write_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037243.227264_291.1","section":"bucket.instance","name":"erivkc-31:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.31","timestamp":"2021-02-11T09:54:03.227264607Z","data":{"read_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":1},"write_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":2},"status":{"status":"write"}}},{"id":"1_1613037243.232856_292.1","section":"bucket.instance","name":"erivkc-31:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.31","timestamp":"2021-02-11T09:54:03.232856122Z","data":{"read_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":1},"write_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037243.287584_293.1","section":"bucket.instance","name":"erivkc-32:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.32","timestamp":"2021-02-11T09:54:03.287584331Z","data":{"read_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":1},"write_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":2},"status":{"status":"write"}}},{"id":"1_1613037243.297721_294.1","section":"bucket.instance","name":"erivkc-32:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.32","timestamp":"2021-02-11T09:54:03.297721301Z","data":{"read_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":1},"write_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037243.370771_295.1","section":"bucket.instance","name":"erivkc-33:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.33","timestamp":"2021-02-11T09:54:03.370771432Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.379000_296.1","section":"bucket.instance","name":"erivkc-33:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.33","timestamp":"2021-02-11T09:54:03.379000661Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.391762_297.1","section":"bucket","name":"erivkc-33","timestamp":"2021-02-11T09:54:03.391762122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ADx5xUaeeeFNdTXMcdfXz-9","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.396407_298.1","section":"bucket","name":"erivkc-33","timestamp":"2021-02-11T09:54:03.396407953Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ADx5xUaeeeFNdTXMcdfXz-9","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.416163_299.1","section":"bucket.instance","name":"erivkc-34:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.34","timestamp":"2021-02-11T09:54:03.416163733Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.423119_300.1","section":"bucket.instance","name":"erivkc-34:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.34","timestamp":"2021-02-11T09:54:03.423119475Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.425169_301.1","section":"bucket","name":"erivkc-34","timestamp":"2021-02-11T09:54:03.425169760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ww9PzUwQW1bSFjXRQ62t6to","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.430515_302.1","section":"bucket","name":"erivkc-34","timestamp":"2021-02-11T09:54:03.430515925Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ww9PzUwQW1bSFjXRQ62t6to","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.472698_303.1","section":"bucket.instance","name":"erivkc-35:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.35","timestamp":"2021-02-11T09:54:03.472698603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.478541_304.1","section":"bucket.instance","name":"erivkc-35:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.35","timestamp":"2021-02-11T09:54:03.478541182Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.480470_305.1","section":"bucket","name":"erivkc-35","timestamp":"2021-02-11T09:54:03.480470578Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Oem2vERyFYbZLeBy-7DQmFi","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.485715_306.1","section":"bucket","name":"erivkc-35","timestamp":"2021-02-11T09:54:03.485715606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Oem2vERyFYbZLeBy-7DQmFi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.508665_307.1","section":"bucket.instance","name":"erivkc-33:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.33","timestamp":"2021-02-11T09:54:03.508665417Z","data":{"read_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":1},"write_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":2},"status":{"status":"write"}}},{"id":"1_1613037243.514139_308.1","section":"bucket.instance","name":"erivkc-33:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.33","timestamp":"2021-02-11T09:54:03.514139374Z","data":{"read_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":1},"write_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037244.562809_309.1","section":"bucket.instance","name":"erivkc-34:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.34","timestamp":"2021-02-11T09:54:04.562809508Z","data":{"read_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":1},"write_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":2},"status":{"status":"write"}}},{"id":"1_1613037244.574097_310.1","section":"bucket.instance","name":"erivkc-34:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.34","timestamp":"2021-02-11T09:54:04.574097270Z","data":{"read_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":1},"write_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037246.158942_311.1","section":"bucket.instance","name":"erivkc-35:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.35","timestamp":"2021-02-11T09:54:06.158942088Z","data":{"read_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":1},"write_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":2},"status":{"status":"write"}}},{"id":"1_1613037246.169566_312.1","section":"bucket.instance","name":"erivkc-35:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.35","timestamp":"2021-02-11T09:54:06.169566208Z","data":{"read_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":1},"write_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037246.707056_313.1","section":"bucket.instance","name":"erivkc-36:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.36","timestamp":"2021-02-11T09:54:06.707056801Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xkVFLPDDAFPFhXQj_hUQN_h","ver":1},"status":{"status":"write"}}},{"id":"1_1613037246.712355_314.1","section":"bucket.instance","name":"erivkc-36:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.36","timestamp":"2021-02-11T09:54:06.712355636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xkVFLPDDAFPFhXQj_hUQN_h","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037246.714533_315.1","section":"bucket","name":"erivkc-36","timestamp":"2021-02-11T09:54:06.714533140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RHYMrzG1L0pGiaLoVMFN6Jh","ver":1},"status":{"status":"write"}}},{"id":"1_1613037246.720239_316.1","section":"bucket","name":"erivkc-36","timestamp":"2021-02-11T09:54:06.720239982Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RHYMrzG1L0pGiaLoVMFN6Jh","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037247.324144_317.1","section":"bucket.instance","name":"erivkc-37:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.37","timestamp":"2021-02-11T09:54:07.324144788Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IULRi3Z7Wy8cbA2hu8UPNbn","ver":1},"status":{"status":"write"}}},{"id":"1_1613037247.329358_318.1","section":"bucket.instance","name":"erivkc-37:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.37","timestamp":"2021-02-11T09:54:07.329358872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IULRi3Z7Wy8cbA2hu8UPNbn","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037247.331412_319.1","section":"bucket","name":"erivkc-37","timestamp":"2021-02-11T09:54:07.331412921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4WsXtBgVYqJH440dvH45LjN","ver":1},"status":{"status":"write"}}},{"id":"1_1613037247.336333_320.1","section":"bucket","name":"erivkc-37","timestamp":"2021-02-11T09:54:07.336333562Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4WsXtBgVYqJH440dvH45LjN","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037248.290625_321.1","section":"bucket.instance","name":"erivkc-38:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.38","timestamp":"2021-02-11T09:54:08.290625366Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HE4UB5TesXvy1aNWgMXkzF_","ver":1},"status":{"status":"write"}}},{"id":"1_1613037248.296282_322.1","section":"bucket.instance","name":"erivkc-38:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.38","timestamp":"2021-02-11T09:54:08.296282179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HE4UB5TesXvy1aNWgMXkzF_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037248.300696_323.1","section":"bucket","name":"erivkc-38","timestamp":"2021-02-11T09:54:08.300696565Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0eGq9xqmjVJXOf9NRs_EZL1","ver":1},"status":{"status":"write"}}},{"id":"1_1613037248.306636_324.1","section":"bucket","name":"erivkc-38","timestamp":"2021-02-11T09:54:08.306636403Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0eGq9xqmjVJXOf9NRs_EZL1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037252.586120_325.1","section":"bucket.instance","name":"erivkc-39:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.39","timestamp":"2021-02-11T09:54:12.586120488Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YrkZR2UOm5zRnAz70C8q0Ts","ver":1},"status":{"status":"write"}}},{"id":"1_1613037252.592030_326.1","section":"bucket.instance","name":"erivkc-39:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.39","timestamp":"2021-02-11T09:54:12.592030725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YrkZR2UOm5zRnAz70C8q0Ts","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037252.594399_327.1","section":"bucket","name":"erivkc-39","timestamp":"2021-02-11T09:54:12.594399304Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bbyzfmKm0nicNsPlFHc7TaY","ver":1},"status":{"status":"write"}}},{"id":"1_1613037252.600603_328.1","section":"bucket","name":"erivkc-39","timestamp":"2021-02-11T09:54:12.600603841Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bbyzfmKm0nicNsPlFHc7TaY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037252.966061_329.1","section":"bucket.instance","name":"erivkc-40:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.40","timestamp":"2021-02-11T09:54:12.966061107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ut4zUpSzQ_IPfFkmOpHBPRo","ver":1},"status":{"status":"write"}}},{"id":"1_1613037252.971049_330.1","section":"bucket.instance","name":"erivkc-40:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.40","timestamp":"2021-02-11T09:54:12.971049650Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ut4zUpSzQ_IPfFkmOpHBPRo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037252.972980_331.1","section":"bucket","name":"erivkc-40","timestamp":"2021-02-11T09:54:12.972980368Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FZ8h3POIHnXIXFvTVd54kUz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037252.978457_332.1","section":"bucket","name":"erivkc-40","timestamp":"2021-02-11T09:54:12.978457985Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FZ8h3POIHnXIXFvTVd54kUz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037253.325406_333.1","section":"bucket.instance","name":"erivkc-41:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.41","timestamp":"2021-02-11T09:54:13.325406285Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l9qP0OWQ5TNNBXmWZGHVzV9","ver":1},"status":{"status":"write"}}},{"id":"1_1613037253.330538_334.1","section":"bucket.instance","name":"erivkc-41:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.41","timestamp":"2021-02-11T09:54:13.330538990Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l9qP0OWQ5TNNBXmWZGHVzV9","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037253.332745_335.1","section":"bucket","name":"erivkc-41","timestamp":"2021-02-11T09:54:13.332745334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hxjfNrnDah4c4Fy16F_1ybu","ver":1},"status":{"status":"write"}}},{"id":"1_1613037253.337510_336.1","section":"bucket","name":"erivkc-41","timestamp":"2021-02-11T09:54:13.337510904Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hxjfNrnDah4c4Fy16F_1ybu","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:56:31.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f2716509-104d-423b-ba5c-db1bd74642ab', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:31.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f2716509-104d-423b-ba5c-db1bd74642ab', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:31.873 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f2716509-104d-423b-ba5c-db1bd74642ab --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:31.911 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:31.911 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:32.181 INFO:teuthology.orchestra.run.smithi133.stdout:[{"id":"1_1613036845.801758_43.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T09:47:25.801758607Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lbhm83ChfGn6EHKoaQzp5TB","ver":1},"status":{"status":"write"}}},{"id":"1_1613036845.807024_48.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T09:47:25.807024959Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lbhm83ChfGn6EHKoaQzp5TB","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036964.781099_55.1","section":"bucket.instance","name":"erivkc-1:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.1","timestamp":"2021-02-11T09:49:24.781099770Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7CX9vbVJ-GdTYgNGNLhgoE1","ver":1},"status":{"status":"write"}}},{"id":"1_1613036964.794012_57.1","section":"bucket.instance","name":"erivkc-1:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.1","timestamp":"2021-02-11T09:49:24.794012217Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7CX9vbVJ-GdTYgNGNLhgoE1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036964.798399_58.1","section":"bucket","name":"erivkc-1","timestamp":"2021-02-11T09:49:24.798399196Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CPPkcG3v2weIBmWYso1zTaV","ver":1},"status":{"status":"write"}}},{"id":"1_1613036964.804170_59.1","section":"bucket","name":"erivkc-1","timestamp":"2021-02-11T09:49:24.804170325Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CPPkcG3v2weIBmWYso1zTaV","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036964.833958_60.1","section":"bucket.instance","name":"erivkc-2:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.2","timestamp":"2021-02-11T09:49:24.833958584Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2FGD8auz1ZhEdsT6v2xSbGp","ver":1},"status":{"status":"write"}}},{"id":"1_1613036964.841848_61.1","section":"bucket.instance","name":"erivkc-2:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.2","timestamp":"2021-02-11T09:49:24.841848223Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2FGD8auz1ZhEdsT6v2xSbGp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036964.843748_62.1","section":"bucket","name":"erivkc-2","timestamp":"2021-02-11T09:49:24.843748711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N0rMsI_g5Yis-zEde5_K0Ir","ver":1},"status":{"status":"write"}}},{"id":"1_1613036964.850242_63.1","section":"bucket","name":"erivkc-2","timestamp":"2021-02-11T09:49:24.850242744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N0rMsI_g5Yis-zEde5_K0Ir","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036968.651442_64.1","section":"bucket.instance","name":"erivkc-3:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.3","timestamp":"2021-02-11T09:49:28.651442746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1S35tybN40BRSEaKI2l3DIk","ver":1},"status":{"status":"write"}}},{"id":"1_1613036968.660175_65.1","section":"bucket.instance","name":"erivkc-3:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.3","timestamp":"2021-02-11T09:49:28.660175151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1S35tybN40BRSEaKI2l3DIk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036968.662383_66.1","section":"bucket","name":"erivkc-3","timestamp":"2021-02-11T09:49:28.662383146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aUbSv47O20-wQgZ_f_BQyws","ver":1},"status":{"status":"write"}}},{"id":"1_1613036968.672641_67.1","section":"bucket","name":"erivkc-3","timestamp":"2021-02-11T09:49:28.672641364Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aUbSv47O20-wQgZ_f_BQyws","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036972.560421_68.1","section":"bucket.instance","name":"erivkc-4:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.4","timestamp":"2021-02-11T09:49:32.560421693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PBJ_0UMaWtCXBaplvwFqFoX","ver":1},"status":{"status":"write"}}},{"id":"1_1613036972.565575_70.1","section":"bucket.instance","name":"erivkc-4:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.4","timestamp":"2021-02-11T09:49:32.565575234Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PBJ_0UMaWtCXBaplvwFqFoX","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036972.567393_71.1","section":"bucket","name":"erivkc-4","timestamp":"2021-02-11T09:49:32.567393622Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1FZcpPqSVgDrFiCSVeHkB-p","ver":1},"status":{"status":"write"}}},{"id":"1_1613036972.572741_72.1","section":"bucket","name":"erivkc-4","timestamp":"2021-02-11T09:49:32.572741163Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1FZcpPqSVgDrFiCSVeHkB-p","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036972.892360_73.1","section":"bucket.instance","name":"erivkc-5:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.5","timestamp":"2021-02-11T09:49:32.892360275Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_97AcbvI_qrWCvEAT4gZODL3","ver":1},"status":{"status":"write"}}},{"id":"1_1613036972.897238_74.1","section":"bucket.instance","name":"erivkc-5:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.5","timestamp":"2021-02-11T09:49:32.897238243Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_97AcbvI_qrWCvEAT4gZODL3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036972.899057_75.1","section":"bucket","name":"erivkc-5","timestamp":"2021-02-11T09:49:32.899057233Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j8qWxRQVbGg7UOusTEwf7B9","ver":1},"status":{"status":"write"}}},{"id":"1_1613036972.904781_76.1","section":"bucket","name":"erivkc-5","timestamp":"2021-02-11T09:49:32.904781579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j8qWxRQVbGg7UOusTEwf7B9","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036973.578464_77.1","section":"bucket.instance","name":"erivkc-6:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.6","timestamp":"2021-02-11T09:49:33.578464448Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y3I0bk2WnnKmeCxS7P_RG4F","ver":1},"status":{"status":"write"}}},{"id":"1_1613036973.583582_78.1","section":"bucket.instance","name":"erivkc-6:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.6","timestamp":"2021-02-11T09:49:33.583582832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y3I0bk2WnnKmeCxS7P_RG4F","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036973.585257_79.1","section":"bucket","name":"erivkc-6","timestamp":"2021-02-11T09:49:33.585257871Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a53snF3YGjGzzmA--K1O94L","ver":1},"status":{"status":"write"}}},{"id":"1_1613036973.589655_80.1","section":"bucket","name":"erivkc-6","timestamp":"2021-02-11T09:49:33.589655678Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a53snF3YGjGzzmA--K1O94L","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.632140_81.1","section":"bucket.instance","name":"erivkc-7:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.7","timestamp":"2021-02-11T09:49:41.632140668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KTbG7jYEk3iEpbVAS1YRyaR","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.637460_84.1","section":"bucket.instance","name":"erivkc-7:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.7","timestamp":"2021-02-11T09:49:41.637460231Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KTbG7jYEk3iEpbVAS1YRyaR","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.640349_85.1","section":"bucket","name":"erivkc-7","timestamp":"2021-02-11T09:49:41.640349109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.647069_86.1","section":"bucket","name":"erivkc-7","timestamp":"2021-02-11T09:49:41.647069461Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.667580_87.1","section":"bucket.instance","name":"erivkc-8:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.8","timestamp":"2021-02-11T09:49:41.667580530Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dg-o8rJKnYjuar4EHq-FKMN","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.675956_88.1","section":"bucket.instance","name":"erivkc-8:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.8","timestamp":"2021-02-11T09:49:41.675956821Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dg-o8rJKnYjuar4EHq-FKMN","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.678555_89.1","section":"bucket","name":"erivkc-8","timestamp":"2021-02-11T09:49:41.678555790Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.684839_90.1","section":"bucket","name":"erivkc-8","timestamp":"2021-02-11T09:49:41.684839859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.745183_91.1","section":"bucket.instance","name":"erivkc-9:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.9","timestamp":"2021-02-11T09:49:41.745183419Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W5J5xaWTI0Gmgb3HWRJ8MbG","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.749950_92.1","section":"bucket.instance","name":"erivkc-9:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.9","timestamp":"2021-02-11T09:49:41.749950640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W5J5xaWTI0Gmgb3HWRJ8MbG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.751718_93.1","section":"bucket","name":"erivkc-9","timestamp":"2021-02-11T09:49:41.751718057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.757312_94.1","section":"bucket","name":"erivkc-9","timestamp":"2021-02-11T09:49:41.757312185Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036984.846663_95.1","section":"bucket","name":"erivkc-7","timestamp":"2021-02-11T09:49:44.846663475Z","data":{"read_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":1},"write_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036984.853017_97.1","section":"bucket","name":"erivkc-7","timestamp":"2021-02-11T09:49:44.853017326Z","data":{"read_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":1},"write_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036984.929328_98.1","section":"bucket","name":"erivkc-8","timestamp":"2021-02-11T09:49:44.929328641Z","data":{"read_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":1},"write_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036984.936781_99.1","section":"bucket","name":"erivkc-8","timestamp":"2021-02-11T09:49:44.936781341Z","data":{"read_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":1},"write_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036985.064812_100.1","section":"bucket","name":"erivkc-9","timestamp":"2021-02-11T09:49:45.064812325Z","data":{"read_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":1},"write_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036985.070217_101.1","section":"bucket","name":"erivkc-9","timestamp":"2021-02-11T09:49:45.070217328Z","data":{"read_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":1},"write_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036999.845976_50.1","section":"bucket.instance","name":"erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10","timestamp":"2021-02-11T09:49:59.845976867Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X61poZNvtY7wornc-IkI9Ue","ver":1},"status":{"status":"write"}}},{"id":"1_1613036999.851424_52.1","section":"bucket.instance","name":"erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10","timestamp":"2021-02-11T09:49:59.851424369Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X61poZNvtY7wornc-IkI9Ue","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036999.853338_53.1","section":"bucket","name":"erivkc-10","timestamp":"2021-02-11T09:49:59.853338236Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f9x9v20LyN2Z56R02tSPZg7","ver":1},"status":{"status":"write"}}},{"id":"1_1613036999.858076_54.1","section":"bucket","name":"erivkc-10","timestamp":"2021-02-11T09:49:59.858076108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f9x9v20LyN2Z56R02tSPZg7","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036999.993526_56.1","section":"bucket.instance","name":"erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11","timestamp":"2021-02-11T09:49:59.993526854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c9r9KX8B4gD8WQmfwj_PhZZ","ver":1},"status":{"status":"write"}}},{"id":"1_1613036999.999923_57.1","section":"bucket.instance","name":"erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11","timestamp":"2021-02-11T09:49:59.999923814Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c9r9KX8B4gD8WQmfwj_PhZZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037000.001777_58.1","section":"bucket","name":"erivkc-11","timestamp":"2021-02-11T09:50:00.001777802Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dqr2cFllRHl99RGY2i9FuEy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037000.007381_59.1","section":"bucket","name":"erivkc-11","timestamp":"2021-02-11T09:50:00.007381054Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dqr2cFllRHl99RGY2i9FuEy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037000.051806_60.1","section":"bucket.instance","name":"erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12","timestamp":"2021-02-11T09:50:00.051806024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D8OCKLAmYRpy1715HaEnUrg","ver":1},"status":{"status":"write"}}},{"id":"1_1613037000.057515_61.1","section":"bucket.instance","name":"erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12","timestamp":"2021-02-11T09:50:00.057515941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D8OCKLAmYRpy1715HaEnUrg","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037000.061137_62.1","section":"bucket","name":"erivkc-12","timestamp":"2021-02-11T09:50:00.061137438Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AkdgcL9F7rK58axzIzhY-5R","ver":1},"status":{"status":"write"}}},{"id":"1_1613037000.074230_63.1","section":"bucket","name":"erivkc-12","timestamp":"2021-02-11T09:50:00.074230334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AkdgcL9F7rK58axzIzhY-5R","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.704912_167.1","section":"bucket.instance","name":"erivkc-13:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.13","timestamp":"2021-02-11T09:50:13.704912580Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wtf4rO_4_UFnmTOV6Ka-w-5","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.710270_168.1","section":"bucket.instance","name":"erivkc-13:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.13","timestamp":"2021-02-11T09:50:13.710270887Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wtf4rO_4_UFnmTOV6Ka-w-5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.712243_169.1","section":"bucket","name":"erivkc-13","timestamp":"2021-02-11T09:50:13.712243833Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xIInLmSguo5ojG9rkSrh6NV","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.717978_170.1","section":"bucket","name":"erivkc-13","timestamp":"2021-02-11T09:50:13.717978080Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xIInLmSguo5ojG9rkSrh6NV","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.741489_171.1","section":"bucket.instance","name":"erivkc-14:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.14","timestamp":"2021-02-11T09:50:13.741489385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qjqHLxJu0bVGtSl_DLxSzVc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.748782_172.1","section":"bucket.instance","name":"erivkc-14:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.14","timestamp":"2021-02-11T09:50:13.748782975Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qjqHLxJu0bVGtSl_DLxSzVc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.750682_173.1","section":"bucket","name":"erivkc-14","timestamp":"2021-02-11T09:50:13.750682310Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w3Y_zBpyOfitYjQtUFfUTuw","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.755278_174.1","section":"bucket","name":"erivkc-14","timestamp":"2021-02-11T09:50:13.755278974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w3Y_zBpyOfitYjQtUFfUTuw","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.798885_175.1","section":"bucket.instance","name":"erivkc-15:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.15","timestamp":"2021-02-11T09:50:13.798885547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8XLPKECf7E8u3tos2GxeuTy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.804452_176.1","section":"bucket.instance","name":"erivkc-15:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.15","timestamp":"2021-02-11T09:50:13.804452901Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8XLPKECf7E8u3tos2GxeuTy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.806359_177.1","section":"bucket","name":"erivkc-15","timestamp":"2021-02-11T09:50:13.806359983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9Prt9QKeINJuoGyCp0FIuyH","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.812144_178.1","section":"bucket","name":"erivkc-15","timestamp":"2021-02-11T09:50:13.812144543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9Prt9QKeINJuoGyCp0FIuyH","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037041.978668_183.1","section":"bucket.instance","name":"erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16","timestamp":"2021-02-11T09:50:41.978668726Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":1},"status":{"status":"write"}}},{"id":"1_1613037041.984135_184.1","section":"bucket.instance","name":"erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16","timestamp":"2021-02-11T09:50:41.984135248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037041.986045_185.1","section":"bucket","name":"erivkc-16","timestamp":"2021-02-11T09:50:41.986045547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RZKX2QLjfnqRLbyJ9ygdT-6","ver":1},"status":{"status":"write"}}},{"id":"1_1613037041.991151_186.1","section":"bucket","name":"erivkc-16","timestamp":"2021-02-11T09:50:41.991151568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RZKX2QLjfnqRLbyJ9ygdT-6","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037042.691423_187.1","section":"bucket.instance","name":"erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17","timestamp":"2021-02-11T09:50:42.691423169Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":1},"status":{"status":"write"}}},{"id":"1_1613037042.696754_188.1","section":"bucket.instance","name":"erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17","timestamp":"2021-02-11T09:50:42.696754644Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037042.698816_189.1","section":"bucket","name":"erivkc-17","timestamp":"2021-02-11T09:50:42.698816857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z7VmmfI938ifTeJ9aE0V1Ml","ver":1},"status":{"status":"write"}}},{"id":"1_1613037042.703975_190.1","section":"bucket","name":"erivkc-17","timestamp":"2021-02-11T09:50:42.703975143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z7VmmfI938ifTeJ9aE0V1Ml","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037043.653339_191.1","section":"bucket.instance","name":"erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18","timestamp":"2021-02-11T09:50:43.653339978Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":1},"status":{"status":"write"}}},{"id":"1_1613037043.659257_192.1","section":"bucket.instance","name":"erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18","timestamp":"2021-02-11T09:50:43.659257779Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037043.661142_193.1","section":"bucket","name":"erivkc-18","timestamp":"2021-02-11T09:50:43.661142206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GjOszKvzPokzjfI2YAb1ciu","ver":1},"status":{"status":"write"}}},{"id":"1_1613037043.666578_194.1","section":"bucket","name":"erivkc-18","timestamp":"2021-02-11T09:50:43.666578075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GjOszKvzPokzjfI2YAb1ciu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037044.503349_195.1","section":"bucket.instance","name":"erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16","timestamp":"2021-02-11T09:50:44.503349589Z","data":{"read_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":1},"write_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":2},"status":{"status":"write"}}},{"id":"1_1613037044.514489_196.1","section":"bucket.instance","name":"erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16","timestamp":"2021-02-11T09:50:44.514489077Z","data":{"read_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":1},"write_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037045.153847_197.1","section":"bucket.instance","name":"erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17","timestamp":"2021-02-11T09:50:45.153847525Z","data":{"read_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":1},"write_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":2},"status":{"status":"write"}}},{"id":"1_1613037045.165645_198.1","section":"bucket.instance","name":"erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17","timestamp":"2021-02-11T09:50:45.165645371Z","data":{"read_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":1},"write_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037045.642925_199.1","section":"bucket.instance","name":"erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18","timestamp":"2021-02-11T09:50:45.642925401Z","data":{"read_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":1},"write_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":2},"status":{"status":"write"}}},{"id":"1_1613037045.651083_200.1","section":"bucket.instance","name":"erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18","timestamp":"2021-02-11T09:50:45.651083022Z","data":{"read_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":1},"write_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037125.481929_201.1","section":"bucket.instance","name":"erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19","timestamp":"2021-02-11T09:52:05.481929899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.487401_202.1","section":"bucket.instance","name":"erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19","timestamp":"2021-02-11T09:52:05.487401571Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.489373_203.1","section":"bucket","name":"erivkc-19","timestamp":"2021-02-11T09:52:05.489373157Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Jff8SGfEPMevwcwi_i-z2vJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.498307_204.1","section":"bucket","name":"erivkc-19","timestamp":"2021-02-11T09:52:05.498307854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Jff8SGfEPMevwcwi_i-z2vJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.505367_205.1","section":"bucket.instance","name":"erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19","timestamp":"2021-02-11T09:52:05.505367585Z","data":{"read_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":1},"write_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":2},"status":{"status":"write"}}},{"id":"1_1613037125.514387_206.1","section":"bucket.instance","name":"erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19","timestamp":"2021-02-11T09:52:05.514387451Z","data":{"read_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":1},"write_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037186.354909_207.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:06.354909614Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.360229_208.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:06.360229344Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037186.362241_209.1","section":"bucket","name":"erivkc-20","timestamp":"2021-02-11T09:53:06.362241066Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f2qv734GBzTTVNMfY72MNwy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.368386_210.1","section":"bucket","name":"erivkc-20","timestamp":"2021-02-11T09:53:06.368386631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f2qv734GBzTTVNMfY72MNwy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037193.931707_211.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:13.931707918Z","data":{"read_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":1},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":2},"status":{"status":"write"}}},{"id":"1_1613037193.942339_212.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:13.942339208Z","data":{"read_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":1},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037200.002098_213.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:20.002098607Z","data":{"read_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":2},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":3},"status":{"status":"write"}}},{"id":"1_1613037200.012526_214.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:20.012526463Z","data":{"read_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":2},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037207.282029_215.1","section":"bucket.instance","name":"erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21","timestamp":"2021-02-11T09:53:27.282029137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.288219_216.1","section":"bucket.instance","name":"erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21","timestamp":"2021-02-11T09:53:27.288219184Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.290439_217.1","section":"bucket","name":"erivkc-21","timestamp":"2021-02-11T09:53:27.290439853Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tvXoIwFs9HpMEOc0iK5QT1h","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.296030_218.1","section":"bucket","name":"erivkc-21","timestamp":"2021-02-11T09:53:27.296030630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tvXoIwFs9HpMEOc0iK5QT1h","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.320765_219.1","section":"bucket.instance","name":"erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22","timestamp":"2021-02-11T09:53:27.320765881Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.326634_220.1","section":"bucket.instance","name":"erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22","timestamp":"2021-02-11T09:53:27.326634616Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.328915_221.1","section":"bucket","name":"erivkc-22","timestamp":"2021-02-11T09:53:27.328915473Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zb993XEJ8BiM8mBe3hvWtO0","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.334920_222.1","section":"bucket","name":"erivkc-22","timestamp":"2021-02-11T09:53:27.334920903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zb993XEJ8BiM8mBe3hvWtO0","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.394575_223.1","section":"bucket.instance","name":"erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23","timestamp":"2021-02-11T09:53:27.394575034Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.403410_224.1","section":"bucket.instance","name":"erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23","timestamp":"2021-02-11T09:53:27.403410972Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.408075_225.1","section":"bucket","name":"erivkc-23","timestamp":"2021-02-11T09:53:27.408075932Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x_zF6jY24d41dOixQ9zJGrE","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.414692_226.1","section":"bucket","name":"erivkc-23","timestamp":"2021-02-11T09:53:27.414692449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x_zF6jY24d41dOixQ9zJGrE","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.445616_227.1","section":"bucket.instance","name":"erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21","timestamp":"2021-02-11T09:53:27.445616320Z","data":{"read_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":1},"write_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":2},"status":{"status":"write"}}},{"id":"1_1613037207.454205_228.1","section":"bucket.instance","name":"erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21","timestamp":"2021-02-11T09:53:27.454205301Z","data":{"read_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":1},"write_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037207.463414_229.1","section":"bucket.instance","name":"erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22","timestamp":"2021-02-11T09:53:27.463414545Z","data":{"read_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":1},"write_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":2},"status":{"status":"write"}}},{"id":"1_1613037207.472168_230.1","section":"bucket.instance","name":"erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22","timestamp":"2021-02-11T09:53:27.472168737Z","data":{"read_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":1},"write_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037207.485112_231.1","section":"bucket.instance","name":"erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23","timestamp":"2021-02-11T09:53:27.485112100Z","data":{"read_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":1},"write_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":2},"status":{"status":"write"}}},{"id":"1_1613037207.496661_232.1","section":"bucket.instance","name":"erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23","timestamp":"2021-02-11T09:53:27.496661758Z","data":{"read_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":1},"write_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037225.179784_233.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.179784590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.187474_234.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.187474262Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.189621_235.1","section":"bucket","name":"erivkc-24","timestamp":"2021-02-11T09:53:45.189621882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KKbtq-SaZp1BbP3NTY0W1Qw","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.196093_236.1","section":"bucket","name":"erivkc-24","timestamp":"2021-02-11T09:53:45.196093466Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KKbtq-SaZp1BbP3NTY0W1Qw","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.222139_237.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.222139168Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.230640_238.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.230640700Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.232853_239.1","section":"bucket","name":"erivkc-25","timestamp":"2021-02-11T09:53:45.232853969Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v3PBdFYuFlkcRuNZcqiRP0s","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.239328_240.1","section":"bucket","name":"erivkc-25","timestamp":"2021-02-11T09:53:45.239328358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v3PBdFYuFlkcRuNZcqiRP0s","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.288768_241.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.288768050Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.296414_242.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.296414547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.298349_243.1","section":"bucket","name":"erivkc-26","timestamp":"2021-02-11T09:53:45.298349270Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3HLmRlz5MeuRgXHTkAYCO5-","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.307308_244.1","section":"bucket","name":"erivkc-26","timestamp":"2021-02-11T09:53:45.307308476Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3HLmRlz5MeuRgXHTkAYCO5-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.338581_245.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.338581131Z","data":{"read_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":1},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":2},"status":{"status":"write"}}},{"id":"1_1613037225.345749_246.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.345749522Z","data":{"read_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":1},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037225.350555_247.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.350555471Z","data":{"read_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":2},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":3},"status":{"status":"write"}}},{"id":"1_1613037225.361606_248.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.361606024Z","data":{"read_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":2},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037225.372379_249.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.372379079Z","data":{"read_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":1},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":2},"status":{"status":"write"}}},{"id":"1_1613037225.386914_250.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.386914935Z","data":{"read_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":1},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037225.397898_251.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.397898539Z","data":{"read_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":2},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":3},"status":{"status":"write"}}},{"id":"1_1613037225.403904_252.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.403904049Z","data":{"read_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":2},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037225.419818_253.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.419818566Z","data":{"read_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":1},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":2},"status":{"status":"write"}}},{"id":"1_1613037225.424682_254.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.424682186Z","data":{"read_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":1},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037225.434889_255.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.434889809Z","data":{"read_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":2},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":3},"status":{"status":"write"}}},{"id":"1_1613037225.441045_256.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.441045732Z","data":{"read_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":2},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037242.662969_257.1","section":"bucket.instance","name":"erivkc-27:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.27","timestamp":"2021-02-11T09:54:02.662969410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.668739_258.1","section":"bucket.instance","name":"erivkc-27:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.27","timestamp":"2021-02-11T09:54:02.668739755Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.670752_259.1","section":"bucket","name":"erivkc-27","timestamp":"2021-02-11T09:54:02.670752769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gH_oydOqixBGu2q9mVl-QS_","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.675376_260.1","section":"bucket","name":"erivkc-27","timestamp":"2021-02-11T09:54:02.675376669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gH_oydOqixBGu2q9mVl-QS_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.700651_261.1","section":"bucket.instance","name":"erivkc-28:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.28","timestamp":"2021-02-11T09:54:02.700651797Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.706443_262.1","section":"bucket.instance","name":"erivkc-28:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.28","timestamp":"2021-02-11T09:54:02.706443830Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.708331_263.1","section":"bucket","name":"erivkc-28","timestamp":"2021-02-11T09:54:02.708331319Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTuUd6ahhMShO2l9OhA78Xc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.713018_264.1","section":"bucket","name":"erivkc-28","timestamp":"2021-02-11T09:54:02.713018965Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTuUd6ahhMShO2l9OhA78Xc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.762579_265.1","section":"bucket.instance","name":"erivkc-29:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.29","timestamp":"2021-02-11T09:54:02.762579200Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.775474_266.1","section":"bucket.instance","name":"erivkc-29:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.29","timestamp":"2021-02-11T09:54:02.775474031Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.782163_267.1","section":"bucket","name":"erivkc-29","timestamp":"2021-02-11T09:54:02.782163490Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OcZh0JgP4psaTeCxR4eUFzB","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.789419_268.1","section":"bucket","name":"erivkc-29","timestamp":"2021-02-11T09:54:02.789419763Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OcZh0JgP4psaTeCxR4eUFzB","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.824967_270.1","section":"bucket.instance","name":"erivkc-27:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.27","timestamp":"2021-02-11T09:54:02.824967755Z","data":{"read_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":1},"write_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":2},"status":{"status":"write"}}},{"id":"1_1613037242.831294_271.1","section":"bucket.instance","name":"erivkc-27:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.27","timestamp":"2021-02-11T09:54:02.831294730Z","data":{"read_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":1},"write_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037242.879129_272.1","section":"bucket.instance","name":"erivkc-28:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.28","timestamp":"2021-02-11T09:54:02.879129288Z","data":{"read_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":1},"write_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":2},"status":{"status":"write"}}},{"id":"1_1613037242.888573_273.1","section":"bucket.instance","name":"erivkc-28:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.28","timestamp":"2021-02-11T09:54:02.888573312Z","data":{"read_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":1},"write_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037242.941315_274.1","section":"bucket.instance","name":"erivkc-29:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.29","timestamp":"2021-02-11T09:54:02.941315096Z","data":{"read_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":1},"write_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":2},"status":{"status":"write"}}},{"id":"1_1613037242.953261_275.1","section":"bucket.instance","name":"erivkc-29:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.29","timestamp":"2021-02-11T09:54:02.953261205Z","data":{"read_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":1},"write_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037243.018597_277.1","section":"bucket.instance","name":"erivkc-30:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.30","timestamp":"2021-02-11T09:54:03.018597532Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.024482_278.1","section":"bucket.instance","name":"erivkc-30:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.30","timestamp":"2021-02-11T09:54:03.024482726Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.026590_279.1","section":"bucket","name":"erivkc-30","timestamp":"2021-02-11T09:54:03.026590985Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b8xacbbSicSxMqmYhFg2pvb","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.032758_280.1","section":"bucket","name":"erivkc-30","timestamp":"2021-02-11T09:54:03.032758243Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b8xacbbSicSxMqmYhFg2pvb","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.060231_281.1","section":"bucket.instance","name":"erivkc-31:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.31","timestamp":"2021-02-11T09:54:03.060231918Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.066319_282.1","section":"bucket.instance","name":"erivkc-31:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.31","timestamp":"2021-02-11T09:54:03.066319997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.068298_283.1","section":"bucket","name":"erivkc-31","timestamp":"2021-02-11T09:54:03.068298381Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DRK-e9sS3ydFf_VBlB5b3Uc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.072941_284.1","section":"bucket","name":"erivkc-31","timestamp":"2021-02-11T09:54:03.072941983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DRK-e9sS3ydFf_VBlB5b3Uc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.112419_285.1","section":"bucket.instance","name":"erivkc-32:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.32","timestamp":"2021-02-11T09:54:03.112419548Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.119240_286.1","section":"bucket.instance","name":"erivkc-32:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.32","timestamp":"2021-02-11T09:54:03.119240251Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.121798_287.1","section":"bucket","name":"erivkc-32","timestamp":"2021-02-11T09:54:03.121798440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_axnp7YglcFxPJLhOhhHHGZS","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.128266_288.1","section":"bucket","name":"erivkc-32","timestamp":"2021-02-11T09:54:03.128266708Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_axnp7YglcFxPJLhOhhHHGZS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.157131_289.1","section":"bucket.instance","name":"erivkc-30:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.30","timestamp":"2021-02-11T09:54:03.157131853Z","data":{"read_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":1},"write_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":2},"status":{"status":"write"}}},{"id":"1_1613037243.168904_290.1","section":"bucket.instance","name":"erivkc-30:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.30","timestamp":"2021-02-11T09:54:03.168904231Z","data":{"read_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":1},"write_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037243.227264_291.1","section":"bucket.instance","name":"erivkc-31:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.31","timestamp":"2021-02-11T09:54:03.227264607Z","data":{"read_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":1},"write_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":2},"status":{"status":"write"}}},{"id":"1_1613037243.232856_292.1","section":"bucket.instance","name":"erivkc-31:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.31","timestamp":"2021-02-11T09:54:03.232856122Z","data":{"read_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":1},"write_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037243.287584_293.1","section":"bucket.instance","name":"erivkc-32:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.32","timestamp":"2021-02-11T09:54:03.287584331Z","data":{"read_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":1},"write_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":2},"status":{"status":"write"}}},{"id":"1_1613037243.297721_294.1","section":"bucket.instance","name":"erivkc-32:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.32","timestamp":"2021-02-11T09:54:03.297721301Z","data":{"read_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":1},"write_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037243.370771_295.1","section":"bucket.instance","name":"erivkc-33:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.33","timestamp":"2021-02-11T09:54:03.370771432Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.379000_296.1","section":"bucket.instance","name":"erivkc-33:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.33","timestamp":"2021-02-11T09:54:03.379000661Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.391762_297.1","section":"bucket","name":"erivkc-33","timestamp":"2021-02-11T09:54:03.391762122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ADx5xUaeeeFNdTXMcdfXz-9","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.396407_298.1","section":"bucket","name":"erivkc-33","timestamp":"2021-02-11T09:54:03.396407953Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ADx5xUaeeeFNdTXMcdfXz-9","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.416163_299.1","section":"bucket.instance","name":"erivkc-34:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.34","timestamp":"2021-02-11T09:54:03.416163733Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.423119_300.1","section":"bucket.instance","name":"erivkc-34:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.34","timestamp":"2021-02-11T09:54:03.423119475Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.425169_301.1","section":"bucket","name":"erivkc-34","timestamp":"2021-02-11T09:54:03.425169760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ww9PzUwQW1bSFjXRQ62t6to","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.430515_302.1","section":"bucket","name":"erivkc-34","timestamp":"2021-02-11T09:54:03.430515925Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ww9PzUwQW1bSFjXRQ62t6to","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.472698_303.1","section":"bucket.instance","name":"erivkc-35:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.35","timestamp":"2021-02-11T09:54:03.472698603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.478541_304.1","section":"bucket.instance","name":"erivkc-35:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.35","timestamp":"2021-02-11T09:54:03.478541182Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.480470_305.1","section":"bucket","name":"erivkc-35","timestamp":"2021-02-11T09:54:03.480470578Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Oem2vERyFYbZLeBy-7DQmFi","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.485715_306.1","section":"bucket","name":"erivkc-35","timestamp":"2021-02-11T09:54:03.485715606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Oem2vERyFYbZLeBy-7DQmFi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.508665_307.1","section":"bucket.instance","name":"erivkc-33:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.33","timestamp":"2021-02-11T09:54:03.508665417Z","data":{"read_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":1},"write_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":2},"status":{"status":"write"}}},{"id":"1_1613037243.514139_308.1","section":"bucket.instance","name":"erivkc-33:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.33","timestamp":"2021-02-11T09:54:03.514139374Z","data":{"read_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":1},"write_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037244.562809_309.1","section":"bucket.instance","name":"erivkc-34:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.34","timestamp":"2021-02-11T09:54:04.562809508Z","data":{"read_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":1},"write_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":2},"status":{"status":"write"}}},{"id":"1_1613037244.574097_310.1","section":"bucket.instance","name":"erivkc-34:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.34","timestamp":"2021-02-11T09:54:04.574097270Z","data":{"read_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":1},"write_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037246.158942_311.1","section":"bucket.instance","name":"erivkc-35:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.35","timestamp":"2021-02-11T09:54:06.158942088Z","data":{"read_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":1},"write_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":2},"status":{"status":"write"}}},{"id":"1_1613037246.169566_312.1","section":"bucket.instance","name":"erivkc-35:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.35","timestamp":"2021-02-11T09:54:06.169566208Z","data":{"read_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":1},"write_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037246.707056_313.1","section":"bucket.instance","name":"erivkc-36:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.36","timestamp":"2021-02-11T09:54:06.707056801Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xkVFLPDDAFPFhXQj_hUQN_h","ver":1},"status":{"status":"write"}}},{"id":"1_1613037246.712355_314.1","section":"bucket.instance","name":"erivkc-36:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.36","timestamp":"2021-02-11T09:54:06.712355636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xkVFLPDDAFPFhXQj_hUQN_h","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037246.714533_315.1","section":"bucket","name":"erivkc-36","timestamp":"2021-02-11T09:54:06.714533140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RHYMrzG1L0pGiaLoVMFN6Jh","ver":1},"status":{"status":"write"}}},{"id":"1_1613037246.720239_316.1","section":"bucket","name":"erivkc-36","timestamp":"2021-02-11T09:54:06.720239982Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RHYMrzG1L0pGiaLoVMFN6Jh","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037247.324144_317.1","section":"bucket.instance","name":"erivkc-37:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.37","timestamp":"2021-02-11T09:54:07.324144788Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IULRi3Z7Wy8cbA2hu8UPNbn","ver":1},"status":{"status":"write"}}},{"id":"1_1613037247.329358_318.1","section":"bucket.instance","name":"erivkc-37:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.37","timestamp":"2021-02-11T09:54:07.329358872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IULRi3Z7Wy8cbA2hu8UPNbn","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037247.331412_319.1","section":"bucket","name":"erivkc-37","timestamp":"2021-02-11T09:54:07.331412921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4WsXtBgVYqJH440dvH45LjN","ver":1},"status":{"status":"write"}}},{"id":"1_1613037247.336333_320.1","section":"bucket","name":"erivkc-37","timestamp":"2021-02-11T09:54:07.336333562Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4WsXtBgVYqJH440dvH45LjN","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037248.290625_321.1","section":"bucket.instance","name":"erivkc-38:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.38","timestamp":"2021-02-11T09:54:08.290625366Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HE4UB5TesXvy1aNWgMXkzF_","ver":1},"status":{"status":"write"}}},{"id":"1_1613037248.296282_322.1","section":"bucket.instance","name":"erivkc-38:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.38","timestamp":"2021-02-11T09:54:08.296282179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HE4UB5TesXvy1aNWgMXkzF_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037248.300696_323.1","section":"bucket","name":"erivkc-38","timestamp":"2021-02-11T09:54:08.300696565Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0eGq9xqmjVJXOf9NRs_EZL1","ver":1},"status":{"status":"write"}}},{"id":"1_1613037248.306636_324.1","section":"bucket","name":"erivkc-38","timestamp":"2021-02-11T09:54:08.306636403Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0eGq9xqmjVJXOf9NRs_EZL1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037252.586120_325.1","section":"bucket.instance","name":"erivkc-39:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.39","timestamp":"2021-02-11T09:54:12.586120488Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YrkZR2UOm5zRnAz70C8q0Ts","ver":1},"status":{"status":"write"}}},{"id":"1_1613037252.592030_326.1","section":"bucket.instance","name":"erivkc-39:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.39","timestamp":"2021-02-11T09:54:12.592030725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YrkZR2UOm5zRnAz70C8q0Ts","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037252.594399_327.1","section":"bucket","name":"erivkc-39","timestamp":"2021-02-11T09:54:12.594399304Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bbyzfmKm0nicNsPlFHc7TaY","ver":1},"status":{"status":"write"}}},{"id":"1_1613037252.600603_328.1","section":"bucket","name":"erivkc-39","timestamp":"2021-02-11T09:54:12.600603841Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bbyzfmKm0nicNsPlFHc7TaY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037252.966061_329.1","section":"bucket.instance","name":"erivkc-40:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.40","timestamp":"2021-02-11T09:54:12.966061107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ut4zUpSzQ_IPfFkmOpHBPRo","ver":1},"status":{"status":"write"}}},{"id":"1_1613037252.971049_330.1","section":"bucket.instance","name":"erivkc-40:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.40","timestamp":"2021-02-11T09:54:12.971049650Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ut4zUpSzQ_IPfFkmOpHBPRo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037252.972980_331.1","section":"bucket","name":"erivkc-40","timestamp":"2021-02-11T09:54:12.972980368Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FZ8h3POIHnXIXFvTVd54kUz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037252.978457_332.1","section":"bucket","name":"erivkc-40","timestamp":"2021-02-11T09:54:12.978457985Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FZ8h3POIHnXIXFvTVd54kUz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037253.325406_333.1","section":"bucket.instance","name":"erivkc-41:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.41","timestamp":"2021-02-11T09:54:13.325406285Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l9qP0OWQ5TNNBXmWZGHVzV9","ver":1},"status":{"status":"write"}}},{"id":"1_1613037253.330538_334.1","section":"bucket.instance","name":"erivkc-41:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.41","timestamp":"2021-02-11T09:54:13.330538990Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l9qP0OWQ5TNNBXmWZGHVzV9","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037253.332745_335.1","section":"bucket","name":"erivkc-41","timestamp":"2021-02-11T09:54:13.332745334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hxjfNrnDah4c4Fy16F_1ybu","ver":1},"status":{"status":"write"}}},{"id":"1_1613037253.337510_336.1","section":"bucket","name":"erivkc-41","timestamp":"2021-02-11T09:54:13.337510904Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hxjfNrnDah4c4Fy16F_1ybu","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:56:32.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f2716509-104d-423b-ba5c-db1bd74642ab', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:32.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f2716509-104d-423b-ba5c-db1bd74642ab', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:32.183 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f2716509-104d-423b-ba5c-db1bd74642ab --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:32.224 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:32.225 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:32.515 INFO:teuthology.orchestra.run.smithi026.stdout:[{"id":"1_1613036845.801758_43.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T09:47:25.801758607Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lbhm83ChfGn6EHKoaQzp5TB","ver":1},"status":{"status":"write"}}},{"id":"1_1613036845.807024_48.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T09:47:25.807024959Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lbhm83ChfGn6EHKoaQzp5TB","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036964.781099_55.1","section":"bucket.instance","name":"erivkc-1:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.1","timestamp":"2021-02-11T09:49:24.781099770Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7CX9vbVJ-GdTYgNGNLhgoE1","ver":1},"status":{"status":"write"}}},{"id":"1_1613036964.794012_57.1","section":"bucket.instance","name":"erivkc-1:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.1","timestamp":"2021-02-11T09:49:24.794012217Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7CX9vbVJ-GdTYgNGNLhgoE1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036964.798399_58.1","section":"bucket","name":"erivkc-1","timestamp":"2021-02-11T09:49:24.798399196Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CPPkcG3v2weIBmWYso1zTaV","ver":1},"status":{"status":"write"}}},{"id":"1_1613036964.804170_59.1","section":"bucket","name":"erivkc-1","timestamp":"2021-02-11T09:49:24.804170325Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CPPkcG3v2weIBmWYso1zTaV","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036964.833958_60.1","section":"bucket.instance","name":"erivkc-2:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.2","timestamp":"2021-02-11T09:49:24.833958584Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2FGD8auz1ZhEdsT6v2xSbGp","ver":1},"status":{"status":"write"}}},{"id":"1_1613036964.841848_61.1","section":"bucket.instance","name":"erivkc-2:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.2","timestamp":"2021-02-11T09:49:24.841848223Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2FGD8auz1ZhEdsT6v2xSbGp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036964.843748_62.1","section":"bucket","name":"erivkc-2","timestamp":"2021-02-11T09:49:24.843748711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N0rMsI_g5Yis-zEde5_K0Ir","ver":1},"status":{"status":"write"}}},{"id":"1_1613036964.850242_63.1","section":"bucket","name":"erivkc-2","timestamp":"2021-02-11T09:49:24.850242744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N0rMsI_g5Yis-zEde5_K0Ir","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036968.651442_64.1","section":"bucket.instance","name":"erivkc-3:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.3","timestamp":"2021-02-11T09:49:28.651442746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1S35tybN40BRSEaKI2l3DIk","ver":1},"status":{"status":"write"}}},{"id":"1_1613036968.660175_65.1","section":"bucket.instance","name":"erivkc-3:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.3","timestamp":"2021-02-11T09:49:28.660175151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1S35tybN40BRSEaKI2l3DIk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036968.662383_66.1","section":"bucket","name":"erivkc-3","timestamp":"2021-02-11T09:49:28.662383146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aUbSv47O20-wQgZ_f_BQyws","ver":1},"status":{"status":"write"}}},{"id":"1_1613036968.672641_67.1","section":"bucket","name":"erivkc-3","timestamp":"2021-02-11T09:49:28.672641364Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aUbSv47O20-wQgZ_f_BQyws","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036972.560421_68.1","section":"bucket.instance","name":"erivkc-4:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.4","timestamp":"2021-02-11T09:49:32.560421693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PBJ_0UMaWtCXBaplvwFqFoX","ver":1},"status":{"status":"write"}}},{"id":"1_1613036972.565575_70.1","section":"bucket.instance","name":"erivkc-4:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.4","timestamp":"2021-02-11T09:49:32.565575234Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PBJ_0UMaWtCXBaplvwFqFoX","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036972.567393_71.1","section":"bucket","name":"erivkc-4","timestamp":"2021-02-11T09:49:32.567393622Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1FZcpPqSVgDrFiCSVeHkB-p","ver":1},"status":{"status":"write"}}},{"id":"1_1613036972.572741_72.1","section":"bucket","name":"erivkc-4","timestamp":"2021-02-11T09:49:32.572741163Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1FZcpPqSVgDrFiCSVeHkB-p","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036972.892360_73.1","section":"bucket.instance","name":"erivkc-5:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.5","timestamp":"2021-02-11T09:49:32.892360275Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_97AcbvI_qrWCvEAT4gZODL3","ver":1},"status":{"status":"write"}}},{"id":"1_1613036972.897238_74.1","section":"bucket.instance","name":"erivkc-5:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.5","timestamp":"2021-02-11T09:49:32.897238243Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_97AcbvI_qrWCvEAT4gZODL3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036972.899057_75.1","section":"bucket","name":"erivkc-5","timestamp":"2021-02-11T09:49:32.899057233Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j8qWxRQVbGg7UOusTEwf7B9","ver":1},"status":{"status":"write"}}},{"id":"1_1613036972.904781_76.1","section":"bucket","name":"erivkc-5","timestamp":"2021-02-11T09:49:32.904781579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j8qWxRQVbGg7UOusTEwf7B9","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036973.578464_77.1","section":"bucket.instance","name":"erivkc-6:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.6","timestamp":"2021-02-11T09:49:33.578464448Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y3I0bk2WnnKmeCxS7P_RG4F","ver":1},"status":{"status":"write"}}},{"id":"1_1613036973.583582_78.1","section":"bucket.instance","name":"erivkc-6:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.6","timestamp":"2021-02-11T09:49:33.583582832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y3I0bk2WnnKmeCxS7P_RG4F","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036973.585257_79.1","section":"bucket","name":"erivkc-6","timestamp":"2021-02-11T09:49:33.585257871Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a53snF3YGjGzzmA--K1O94L","ver":1},"status":{"status":"write"}}},{"id":"1_1613036973.589655_80.1","section":"bucket","name":"erivkc-6","timestamp":"2021-02-11T09:49:33.589655678Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a53snF3YGjGzzmA--K1O94L","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.632140_81.1","section":"bucket.instance","name":"erivkc-7:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.7","timestamp":"2021-02-11T09:49:41.632140668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KTbG7jYEk3iEpbVAS1YRyaR","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.637460_84.1","section":"bucket.instance","name":"erivkc-7:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.7","timestamp":"2021-02-11T09:49:41.637460231Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KTbG7jYEk3iEpbVAS1YRyaR","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.640349_85.1","section":"bucket","name":"erivkc-7","timestamp":"2021-02-11T09:49:41.640349109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.647069_86.1","section":"bucket","name":"erivkc-7","timestamp":"2021-02-11T09:49:41.647069461Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.667580_87.1","section":"bucket.instance","name":"erivkc-8:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.8","timestamp":"2021-02-11T09:49:41.667580530Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dg-o8rJKnYjuar4EHq-FKMN","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.675956_88.1","section":"bucket.instance","name":"erivkc-8:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.8","timestamp":"2021-02-11T09:49:41.675956821Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dg-o8rJKnYjuar4EHq-FKMN","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.678555_89.1","section":"bucket","name":"erivkc-8","timestamp":"2021-02-11T09:49:41.678555790Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.684839_90.1","section":"bucket","name":"erivkc-8","timestamp":"2021-02-11T09:49:41.684839859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.745183_91.1","section":"bucket.instance","name":"erivkc-9:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.9","timestamp":"2021-02-11T09:49:41.745183419Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W5J5xaWTI0Gmgb3HWRJ8MbG","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.749950_92.1","section":"bucket.instance","name":"erivkc-9:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.9","timestamp":"2021-02-11T09:49:41.749950640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W5J5xaWTI0Gmgb3HWRJ8MbG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.751718_93.1","section":"bucket","name":"erivkc-9","timestamp":"2021-02-11T09:49:41.751718057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.757312_94.1","section":"bucket","name":"erivkc-9","timestamp":"2021-02-11T09:49:41.757312185Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036984.846663_95.1","section":"bucket","name":"erivkc-7","timestamp":"2021-02-11T09:49:44.846663475Z","data":{"read_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":1},"write_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036984.853017_97.1","section":"bucket","name":"erivkc-7","timestamp":"2021-02-11T09:49:44.853017326Z","data":{"read_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":1},"write_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036984.929328_98.1","section":"bucket","name":"erivkc-8","timestamp":"2021-02-11T09:49:44.929328641Z","data":{"read_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":1},"write_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036984.936781_99.1","section":"bucket","name":"erivkc-8","timestamp":"2021-02-11T09:49:44.936781341Z","data":{"read_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":1},"write_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036985.064812_100.1","section":"bucket","name":"erivkc-9","timestamp":"2021-02-11T09:49:45.064812325Z","data":{"read_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":1},"write_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036985.070217_101.1","section":"bucket","name":"erivkc-9","timestamp":"2021-02-11T09:49:45.070217328Z","data":{"read_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":1},"write_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036999.845976_50.1","section":"bucket.instance","name":"erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10","timestamp":"2021-02-11T09:49:59.845976867Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X61poZNvtY7wornc-IkI9Ue","ver":1},"status":{"status":"write"}}},{"id":"1_1613036999.851424_52.1","section":"bucket.instance","name":"erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10","timestamp":"2021-02-11T09:49:59.851424369Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X61poZNvtY7wornc-IkI9Ue","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036999.853338_53.1","section":"bucket","name":"erivkc-10","timestamp":"2021-02-11T09:49:59.853338236Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f9x9v20LyN2Z56R02tSPZg7","ver":1},"status":{"status":"write"}}},{"id":"1_1613036999.858076_54.1","section":"bucket","name":"erivkc-10","timestamp":"2021-02-11T09:49:59.858076108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f9x9v20LyN2Z56R02tSPZg7","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036999.993526_56.1","section":"bucket.instance","name":"erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11","timestamp":"2021-02-11T09:49:59.993526854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c9r9KX8B4gD8WQmfwj_PhZZ","ver":1},"status":{"status":"write"}}},{"id":"1_1613036999.999923_57.1","section":"bucket.instance","name":"erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11","timestamp":"2021-02-11T09:49:59.999923814Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c9r9KX8B4gD8WQmfwj_PhZZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037000.001777_58.1","section":"bucket","name":"erivkc-11","timestamp":"2021-02-11T09:50:00.001777802Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dqr2cFllRHl99RGY2i9FuEy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037000.007381_59.1","section":"bucket","name":"erivkc-11","timestamp":"2021-02-11T09:50:00.007381054Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dqr2cFllRHl99RGY2i9FuEy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037000.051806_60.1","section":"bucket.instance","name":"erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12","timestamp":"2021-02-11T09:50:00.051806024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D8OCKLAmYRpy1715HaEnUrg","ver":1},"status":{"status":"write"}}},{"id":"1_1613037000.057515_61.1","section":"bucket.instance","name":"erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12","timestamp":"2021-02-11T09:50:00.057515941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D8OCKLAmYRpy1715HaEnUrg","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037000.061137_62.1","section":"bucket","name":"erivkc-12","timestamp":"2021-02-11T09:50:00.061137438Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AkdgcL9F7rK58axzIzhY-5R","ver":1},"status":{"status":"write"}}},{"id":"1_1613037000.074230_63.1","section":"bucket","name":"erivkc-12","timestamp":"2021-02-11T09:50:00.074230334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AkdgcL9F7rK58axzIzhY-5R","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.704912_167.1","section":"bucket.instance","name":"erivkc-13:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.13","timestamp":"2021-02-11T09:50:13.704912580Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wtf4rO_4_UFnmTOV6Ka-w-5","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.710270_168.1","section":"bucket.instance","name":"erivkc-13:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.13","timestamp":"2021-02-11T09:50:13.710270887Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wtf4rO_4_UFnmTOV6Ka-w-5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.712243_169.1","section":"bucket","name":"erivkc-13","timestamp":"2021-02-11T09:50:13.712243833Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xIInLmSguo5ojG9rkSrh6NV","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.717978_170.1","section":"bucket","name":"erivkc-13","timestamp":"2021-02-11T09:50:13.717978080Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xIInLmSguo5ojG9rkSrh6NV","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.741489_171.1","section":"bucket.instance","name":"erivkc-14:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.14","timestamp":"2021-02-11T09:50:13.741489385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qjqHLxJu0bVGtSl_DLxSzVc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.748782_172.1","section":"bucket.instance","name":"erivkc-14:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.14","timestamp":"2021-02-11T09:50:13.748782975Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qjqHLxJu0bVGtSl_DLxSzVc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.750682_173.1","section":"bucket","name":"erivkc-14","timestamp":"2021-02-11T09:50:13.750682310Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w3Y_zBpyOfitYjQtUFfUTuw","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.755278_174.1","section":"bucket","name":"erivkc-14","timestamp":"2021-02-11T09:50:13.755278974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w3Y_zBpyOfitYjQtUFfUTuw","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.798885_175.1","section":"bucket.instance","name":"erivkc-15:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.15","timestamp":"2021-02-11T09:50:13.798885547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8XLPKECf7E8u3tos2GxeuTy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.804452_176.1","section":"bucket.instance","name":"erivkc-15:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.15","timestamp":"2021-02-11T09:50:13.804452901Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8XLPKECf7E8u3tos2GxeuTy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.806359_177.1","section":"bucket","name":"erivkc-15","timestamp":"2021-02-11T09:50:13.806359983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9Prt9QKeINJuoGyCp0FIuyH","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.812144_178.1","section":"bucket","name":"erivkc-15","timestamp":"2021-02-11T09:50:13.812144543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9Prt9QKeINJuoGyCp0FIuyH","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037041.978668_183.1","section":"bucket.instance","name":"erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16","timestamp":"2021-02-11T09:50:41.978668726Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":1},"status":{"status":"write"}}},{"id":"1_1613037041.984135_184.1","section":"bucket.instance","name":"erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16","timestamp":"2021-02-11T09:50:41.984135248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037041.986045_185.1","section":"bucket","name":"erivkc-16","timestamp":"2021-02-11T09:50:41.986045547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RZKX2QLjfnqRLbyJ9ygdT-6","ver":1},"status":{"status":"write"}}},{"id":"1_1613037041.991151_186.1","section":"bucket","name":"erivkc-16","timestamp":"2021-02-11T09:50:41.991151568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RZKX2QLjfnqRLbyJ9ygdT-6","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037042.691423_187.1","section":"bucket.instance","name":"erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17","timestamp":"2021-02-11T09:50:42.691423169Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":1},"status":{"status":"write"}}},{"id":"1_1613037042.696754_188.1","section":"bucket.instance","name":"erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17","timestamp":"2021-02-11T09:50:42.696754644Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037042.698816_189.1","section":"bucket","name":"erivkc-17","timestamp":"2021-02-11T09:50:42.698816857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z7VmmfI938ifTeJ9aE0V1Ml","ver":1},"status":{"status":"write"}}},{"id":"1_1613037042.703975_190.1","section":"bucket","name":"erivkc-17","timestamp":"2021-02-11T09:50:42.703975143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z7VmmfI938ifTeJ9aE0V1Ml","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037043.653339_191.1","section":"bucket.instance","name":"erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18","timestamp":"2021-02-11T09:50:43.653339978Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":1},"status":{"status":"write"}}},{"id":"1_1613037043.659257_192.1","section":"bucket.instance","name":"erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18","timestamp":"2021-02-11T09:50:43.659257779Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037043.661142_193.1","section":"bucket","name":"erivkc-18","timestamp":"2021-02-11T09:50:43.661142206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GjOszKvzPokzjfI2YAb1ciu","ver":1},"status":{"status":"write"}}},{"id":"1_1613037043.666578_194.1","section":"bucket","name":"erivkc-18","timestamp":"2021-02-11T09:50:43.666578075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GjOszKvzPokzjfI2YAb1ciu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037044.503349_195.1","section":"bucket.instance","name":"erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16","timestamp":"2021-02-11T09:50:44.503349589Z","data":{"read_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":1},"write_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":2},"status":{"status":"write"}}},{"id":"1_1613037044.514489_196.1","section":"bucket.instance","name":"erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16","timestamp":"2021-02-11T09:50:44.514489077Z","data":{"read_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":1},"write_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037045.153847_197.1","section":"bucket.instance","name":"erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17","timestamp":"2021-02-11T09:50:45.153847525Z","data":{"read_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":1},"write_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":2},"status":{"status":"write"}}},{"id":"1_1613037045.165645_198.1","section":"bucket.instance","name":"erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17","timestamp":"2021-02-11T09:50:45.165645371Z","data":{"read_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":1},"write_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037045.642925_199.1","section":"bucket.instance","name":"erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18","timestamp":"2021-02-11T09:50:45.642925401Z","data":{"read_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":1},"write_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":2},"status":{"status":"write"}}},{"id":"1_1613037045.651083_200.1","section":"bucket.instance","name":"erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18","timestamp":"2021-02-11T09:50:45.651083022Z","data":{"read_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":1},"write_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037125.481929_201.1","section":"bucket.instance","name":"erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19","timestamp":"2021-02-11T09:52:05.481929899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.487401_202.1","section":"bucket.instance","name":"erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19","timestamp":"2021-02-11T09:52:05.487401571Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.489373_203.1","section":"bucket","name":"erivkc-19","timestamp":"2021-02-11T09:52:05.489373157Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Jff8SGfEPMevwcwi_i-z2vJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.498307_204.1","section":"bucket","name":"erivkc-19","timestamp":"2021-02-11T09:52:05.498307854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Jff8SGfEPMevwcwi_i-z2vJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.505367_205.1","section":"bucket.instance","name":"erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19","timestamp":"2021-02-11T09:52:05.505367585Z","data":{"read_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":1},"write_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":2},"status":{"status":"write"}}},{"id":"1_1613037125.514387_206.1","section":"bucket.instance","name":"erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19","timestamp":"2021-02-11T09:52:05.514387451Z","data":{"read_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":1},"write_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037186.354909_207.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:06.354909614Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.360229_208.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:06.360229344Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037186.362241_209.1","section":"bucket","name":"erivkc-20","timestamp":"2021-02-11T09:53:06.362241066Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f2qv734GBzTTVNMfY72MNwy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.368386_210.1","section":"bucket","name":"erivkc-20","timestamp":"2021-02-11T09:53:06.368386631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f2qv734GBzTTVNMfY72MNwy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037193.931707_211.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:13.931707918Z","data":{"read_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":1},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":2},"status":{"status":"write"}}},{"id":"1_1613037193.942339_212.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:13.942339208Z","data":{"read_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":1},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037200.002098_213.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:20.002098607Z","data":{"read_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":2},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":3},"status":{"status":"write"}}},{"id":"1_1613037200.012526_214.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:20.012526463Z","data":{"read_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":2},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037207.282029_215.1","section":"bucket.instance","name":"erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21","timestamp":"2021-02-11T09:53:27.282029137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.288219_216.1","section":"bucket.instance","name":"erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21","timestamp":"2021-02-11T09:53:27.288219184Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.290439_217.1","section":"bucket","name":"erivkc-21","timestamp":"2021-02-11T09:53:27.290439853Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tvXoIwFs9HpMEOc0iK5QT1h","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.296030_218.1","section":"bucket","name":"erivkc-21","timestamp":"2021-02-11T09:53:27.296030630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tvXoIwFs9HpMEOc0iK5QT1h","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.320765_219.1","section":"bucket.instance","name":"erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22","timestamp":"2021-02-11T09:53:27.320765881Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.326634_220.1","section":"bucket.instance","name":"erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22","timestamp":"2021-02-11T09:53:27.326634616Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.328915_221.1","section":"bucket","name":"erivkc-22","timestamp":"2021-02-11T09:53:27.328915473Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zb993XEJ8BiM8mBe3hvWtO0","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.334920_222.1","section":"bucket","name":"erivkc-22","timestamp":"2021-02-11T09:53:27.334920903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zb993XEJ8BiM8mBe3hvWtO0","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.394575_223.1","section":"bucket.instance","name":"erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23","timestamp":"2021-02-11T09:53:27.394575034Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.403410_224.1","section":"bucket.instance","name":"erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23","timestamp":"2021-02-11T09:53:27.403410972Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.408075_225.1","section":"bucket","name":"erivkc-23","timestamp":"2021-02-11T09:53:27.408075932Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x_zF6jY24d41dOixQ9zJGrE","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.414692_226.1","section":"bucket","name":"erivkc-23","timestamp":"2021-02-11T09:53:27.414692449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x_zF6jY24d41dOixQ9zJGrE","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.445616_227.1","section":"bucket.instance","name":"erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21","timestamp":"2021-02-11T09:53:27.445616320Z","data":{"read_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":1},"write_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":2},"status":{"status":"write"}}},{"id":"1_1613037207.454205_228.1","section":"bucket.instance","name":"erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21","timestamp":"2021-02-11T09:53:27.454205301Z","data":{"read_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":1},"write_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037207.463414_229.1","section":"bucket.instance","name":"erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22","timestamp":"2021-02-11T09:53:27.463414545Z","data":{"read_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":1},"write_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":2},"status":{"status":"write"}}},{"id":"1_1613037207.472168_230.1","section":"bucket.instance","name":"erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22","timestamp":"2021-02-11T09:53:27.472168737Z","data":{"read_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":1},"write_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037207.485112_231.1","section":"bucket.instance","name":"erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23","timestamp":"2021-02-11T09:53:27.485112100Z","data":{"read_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":1},"write_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":2},"status":{"status":"write"}}},{"id":"1_1613037207.496661_232.1","section":"bucket.instance","name":"erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23","timestamp":"2021-02-11T09:53:27.496661758Z","data":{"read_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":1},"write_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037225.179784_233.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.179784590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.187474_234.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.187474262Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.189621_235.1","section":"bucket","name":"erivkc-24","timestamp":"2021-02-11T09:53:45.189621882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KKbtq-SaZp1BbP3NTY0W1Qw","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.196093_236.1","section":"bucket","name":"erivkc-24","timestamp":"2021-02-11T09:53:45.196093466Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KKbtq-SaZp1BbP3NTY0W1Qw","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.222139_237.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.222139168Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.230640_238.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.230640700Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.232853_239.1","section":"bucket","name":"erivkc-25","timestamp":"2021-02-11T09:53:45.232853969Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v3PBdFYuFlkcRuNZcqiRP0s","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.239328_240.1","section":"bucket","name":"erivkc-25","timestamp":"2021-02-11T09:53:45.239328358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v3PBdFYuFlkcRuNZcqiRP0s","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.288768_241.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.288768050Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.296414_242.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.296414547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.298349_243.1","section":"bucket","name":"erivkc-26","timestamp":"2021-02-11T09:53:45.298349270Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3HLmRlz5MeuRgXHTkAYCO5-","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.307308_244.1","section":"bucket","name":"erivkc-26","timestamp":"2021-02-11T09:53:45.307308476Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3HLmRlz5MeuRgXHTkAYCO5-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.338581_245.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.338581131Z","data":{"read_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":1},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":2},"status":{"status":"write"}}},{"id":"1_1613037225.345749_246.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.345749522Z","data":{"read_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":1},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037225.350555_247.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.350555471Z","data":{"read_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":2},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":3},"status":{"status":"write"}}},{"id":"1_1613037225.361606_248.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.361606024Z","data":{"read_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":2},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037225.372379_249.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.372379079Z","data":{"read_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":1},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":2},"status":{"status":"write"}}},{"id":"1_1613037225.386914_250.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.386914935Z","data":{"read_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":1},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037225.397898_251.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.397898539Z","data":{"read_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":2},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":3},"status":{"status":"write"}}},{"id":"1_1613037225.403904_252.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.403904049Z","data":{"read_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":2},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037225.419818_253.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.419818566Z","data":{"read_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":1},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":2},"status":{"status":"write"}}},{"id":"1_1613037225.424682_254.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.424682186Z","data":{"read_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":1},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037225.434889_255.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.434889809Z","data":{"read_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":2},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":3},"status":{"status":"write"}}},{"id":"1_1613037225.441045_256.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.441045732Z","data":{"read_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":2},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037242.662969_257.1","section":"bucket.instance","name":"erivkc-27:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.27","timestamp":"2021-02-11T09:54:02.662969410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.668739_258.1","section":"bucket.instance","name":"erivkc-27:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.27","timestamp":"2021-02-11T09:54:02.668739755Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.670752_259.1","section":"bucket","name":"erivkc-27","timestamp":"2021-02-11T09:54:02.670752769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gH_oydOqixBGu2q9mVl-QS_","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.675376_260.1","section":"bucket","name":"erivkc-27","timestamp":"2021-02-11T09:54:02.675376669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gH_oydOqixBGu2q9mVl-QS_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.700651_261.1","section":"bucket.instance","name":"erivkc-28:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.28","timestamp":"2021-02-11T09:54:02.700651797Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.706443_262.1","section":"bucket.instance","name":"erivkc-28:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.28","timestamp":"2021-02-11T09:54:02.706443830Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.708331_263.1","section":"bucket","name":"erivkc-28","timestamp":"2021-02-11T09:54:02.708331319Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTuUd6ahhMShO2l9OhA78Xc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.713018_264.1","section":"bucket","name":"erivkc-28","timestamp":"2021-02-11T09:54:02.713018965Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTuUd6ahhMShO2l9OhA78Xc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.762579_265.1","section":"bucket.instance","name":"erivkc-29:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.29","timestamp":"2021-02-11T09:54:02.762579200Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.775474_266.1","section":"bucket.instance","name":"erivkc-29:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.29","timestamp":"2021-02-11T09:54:02.775474031Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.782163_267.1","section":"bucket","name":"erivkc-29","timestamp":"2021-02-11T09:54:02.782163490Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OcZh0JgP4psaTeCxR4eUFzB","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.789419_268.1","section":"bucket","name":"erivkc-29","timestamp":"2021-02-11T09:54:02.789419763Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OcZh0JgP4psaTeCxR4eUFzB","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.824967_270.1","section":"bucket.instance","name":"erivkc-27:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.27","timestamp":"2021-02-11T09:54:02.824967755Z","data":{"read_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":1},"write_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":2},"status":{"status":"write"}}},{"id":"1_1613037242.831294_271.1","section":"bucket.instance","name":"erivkc-27:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.27","timestamp":"2021-02-11T09:54:02.831294730Z","data":{"read_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":1},"write_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037242.879129_272.1","section":"bucket.instance","name":"erivkc-28:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.28","timestamp":"2021-02-11T09:54:02.879129288Z","data":{"read_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":1},"write_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":2},"status":{"status":"write"}}},{"id":"1_1613037242.888573_273.1","section":"bucket.instance","name":"erivkc-28:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.28","timestamp":"2021-02-11T09:54:02.888573312Z","data":{"read_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":1},"write_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037242.941315_274.1","section":"bucket.instance","name":"erivkc-29:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.29","timestamp":"2021-02-11T09:54:02.941315096Z","data":{"read_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":1},"write_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":2},"status":{"status":"write"}}},{"id":"1_1613037242.953261_275.1","section":"bucket.instance","name":"erivkc-29:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.29","timestamp":"2021-02-11T09:54:02.953261205Z","data":{"read_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":1},"write_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037243.018597_277.1","section":"bucket.instance","name":"erivkc-30:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.30","timestamp":"2021-02-11T09:54:03.018597532Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.024482_278.1","section":"bucket.instance","name":"erivkc-30:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.30","timestamp":"2021-02-11T09:54:03.024482726Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.026590_279.1","section":"bucket","name":"erivkc-30","timestamp":"2021-02-11T09:54:03.026590985Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b8xacbbSicSxMqmYhFg2pvb","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.032758_280.1","section":"bucket","name":"erivkc-30","timestamp":"2021-02-11T09:54:03.032758243Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b8xacbbSicSxMqmYhFg2pvb","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.060231_281.1","section":"bucket.instance","name":"erivkc-31:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.31","timestamp":"2021-02-11T09:54:03.060231918Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.066319_282.1","section":"bucket.instance","name":"erivkc-31:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.31","timestamp":"2021-02-11T09:54:03.066319997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.068298_283.1","section":"bucket","name":"erivkc-31","timestamp":"2021-02-11T09:54:03.068298381Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DRK-e9sS3ydFf_VBlB5b3Uc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.072941_284.1","section":"bucket","name":"erivkc-31","timestamp":"2021-02-11T09:54:03.072941983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DRK-e9sS3ydFf_VBlB5b3Uc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.112419_285.1","section":"bucket.instance","name":"erivkc-32:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.32","timestamp":"2021-02-11T09:54:03.112419548Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.119240_286.1","section":"bucket.instance","name":"erivkc-32:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.32","timestamp":"2021-02-11T09:54:03.119240251Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.121798_287.1","section":"bucket","name":"erivkc-32","timestamp":"2021-02-11T09:54:03.121798440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_axnp7YglcFxPJLhOhhHHGZS","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.128266_288.1","section":"bucket","name":"erivkc-32","timestamp":"2021-02-11T09:54:03.128266708Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_axnp7YglcFxPJLhOhhHHGZS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.157131_289.1","section":"bucket.instance","name":"erivkc-30:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.30","timestamp":"2021-02-11T09:54:03.157131853Z","data":{"read_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":1},"write_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":2},"status":{"status":"write"}}},{"id":"1_1613037243.168904_290.1","section":"bucket.instance","name":"erivkc-30:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.30","timestamp":"2021-02-11T09:54:03.168904231Z","data":{"read_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":1},"write_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037243.227264_291.1","section":"bucket.instance","name":"erivkc-31:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.31","timestamp":"2021-02-11T09:54:03.227264607Z","data":{"read_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":1},"write_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":2},"status":{"status":"write"}}},{"id":"1_1613037243.232856_292.1","section":"bucket.instance","name":"erivkc-31:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.31","timestamp":"2021-02-11T09:54:03.232856122Z","data":{"read_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":1},"write_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037243.287584_293.1","section":"bucket.instance","name":"erivkc-32:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.32","timestamp":"2021-02-11T09:54:03.287584331Z","data":{"read_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":1},"write_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":2},"status":{"status":"write"}}},{"id":"1_1613037243.297721_294.1","section":"bucket.instance","name":"erivkc-32:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.32","timestamp":"2021-02-11T09:54:03.297721301Z","data":{"read_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":1},"write_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037243.370771_295.1","section":"bucket.instance","name":"erivkc-33:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.33","timestamp":"2021-02-11T09:54:03.370771432Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.379000_296.1","section":"bucket.instance","name":"erivkc-33:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.33","timestamp":"2021-02-11T09:54:03.379000661Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.391762_297.1","section":"bucket","name":"erivkc-33","timestamp":"2021-02-11T09:54:03.391762122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ADx5xUaeeeFNdTXMcdfXz-9","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.396407_298.1","section":"bucket","name":"erivkc-33","timestamp":"2021-02-11T09:54:03.396407953Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ADx5xUaeeeFNdTXMcdfXz-9","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.416163_299.1","section":"bucket.instance","name":"erivkc-34:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.34","timestamp":"2021-02-11T09:54:03.416163733Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.423119_300.1","section":"bucket.instance","name":"erivkc-34:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.34","timestamp":"2021-02-11T09:54:03.423119475Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.425169_301.1","section":"bucket","name":"erivkc-34","timestamp":"2021-02-11T09:54:03.425169760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ww9PzUwQW1bSFjXRQ62t6to","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.430515_302.1","section":"bucket","name":"erivkc-34","timestamp":"2021-02-11T09:54:03.430515925Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ww9PzUwQW1bSFjXRQ62t6to","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.472698_303.1","section":"bucket.instance","name":"erivkc-35:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.35","timestamp":"2021-02-11T09:54:03.472698603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.478541_304.1","section":"bucket.instance","name":"erivkc-35:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.35","timestamp":"2021-02-11T09:54:03.478541182Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.480470_305.1","section":"bucket","name":"erivkc-35","timestamp":"2021-02-11T09:54:03.480470578Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Oem2vERyFYbZLeBy-7DQmFi","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.485715_306.1","section":"bucket","name":"erivkc-35","timestamp":"2021-02-11T09:54:03.485715606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Oem2vERyFYbZLeBy-7DQmFi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.508665_307.1","section":"bucket.instance","name":"erivkc-33:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.33","timestamp":"2021-02-11T09:54:03.508665417Z","data":{"read_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":1},"write_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":2},"status":{"status":"write"}}},{"id":"1_1613037243.514139_308.1","section":"bucket.instance","name":"erivkc-33:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.33","timestamp":"2021-02-11T09:54:03.514139374Z","data":{"read_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":1},"write_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037244.562809_309.1","section":"bucket.instance","name":"erivkc-34:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.34","timestamp":"2021-02-11T09:54:04.562809508Z","data":{"read_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":1},"write_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":2},"status":{"status":"write"}}},{"id":"1_1613037244.574097_310.1","section":"bucket.instance","name":"erivkc-34:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.34","timestamp":"2021-02-11T09:54:04.574097270Z","data":{"read_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":1},"write_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037246.158942_311.1","section":"bucket.instance","name":"erivkc-35:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.35","timestamp":"2021-02-11T09:54:06.158942088Z","data":{"read_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":1},"write_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":2},"status":{"status":"write"}}},{"id":"1_1613037246.169566_312.1","section":"bucket.instance","name":"erivkc-35:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.35","timestamp":"2021-02-11T09:54:06.169566208Z","data":{"read_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":1},"write_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037246.707056_313.1","section":"bucket.instance","name":"erivkc-36:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.36","timestamp":"2021-02-11T09:54:06.707056801Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xkVFLPDDAFPFhXQj_hUQN_h","ver":1},"status":{"status":"write"}}},{"id":"1_1613037246.712355_314.1","section":"bucket.instance","name":"erivkc-36:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.36","timestamp":"2021-02-11T09:54:06.712355636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xkVFLPDDAFPFhXQj_hUQN_h","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037246.714533_315.1","section":"bucket","name":"erivkc-36","timestamp":"2021-02-11T09:54:06.714533140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RHYMrzG1L0pGiaLoVMFN6Jh","ver":1},"status":{"status":"write"}}},{"id":"1_1613037246.720239_316.1","section":"bucket","name":"erivkc-36","timestamp":"2021-02-11T09:54:06.720239982Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RHYMrzG1L0pGiaLoVMFN6Jh","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037247.324144_317.1","section":"bucket.instance","name":"erivkc-37:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.37","timestamp":"2021-02-11T09:54:07.324144788Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IULRi3Z7Wy8cbA2hu8UPNbn","ver":1},"status":{"status":"write"}}},{"id":"1_1613037247.329358_318.1","section":"bucket.instance","name":"erivkc-37:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.37","timestamp":"2021-02-11T09:54:07.329358872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IULRi3Z7Wy8cbA2hu8UPNbn","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037247.331412_319.1","section":"bucket","name":"erivkc-37","timestamp":"2021-02-11T09:54:07.331412921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4WsXtBgVYqJH440dvH45LjN","ver":1},"status":{"status":"write"}}},{"id":"1_1613037247.336333_320.1","section":"bucket","name":"erivkc-37","timestamp":"2021-02-11T09:54:07.336333562Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4WsXtBgVYqJH440dvH45LjN","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037248.290625_321.1","section":"bucket.instance","name":"erivkc-38:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.38","timestamp":"2021-02-11T09:54:08.290625366Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HE4UB5TesXvy1aNWgMXkzF_","ver":1},"status":{"status":"write"}}},{"id":"1_1613037248.296282_322.1","section":"bucket.instance","name":"erivkc-38:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.38","timestamp":"2021-02-11T09:54:08.296282179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HE4UB5TesXvy1aNWgMXkzF_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037248.300696_323.1","section":"bucket","name":"erivkc-38","timestamp":"2021-02-11T09:54:08.300696565Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0eGq9xqmjVJXOf9NRs_EZL1","ver":1},"status":{"status":"write"}}},{"id":"1_1613037248.306636_324.1","section":"bucket","name":"erivkc-38","timestamp":"2021-02-11T09:54:08.306636403Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0eGq9xqmjVJXOf9NRs_EZL1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037252.586120_325.1","section":"bucket.instance","name":"erivkc-39:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.39","timestamp":"2021-02-11T09:54:12.586120488Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YrkZR2UOm5zRnAz70C8q0Ts","ver":1},"status":{"status":"write"}}},{"id":"1_1613037252.592030_326.1","section":"bucket.instance","name":"erivkc-39:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.39","timestamp":"2021-02-11T09:54:12.592030725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YrkZR2UOm5zRnAz70C8q0Ts","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037252.594399_327.1","section":"bucket","name":"erivkc-39","timestamp":"2021-02-11T09:54:12.594399304Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bbyzfmKm0nicNsPlFHc7TaY","ver":1},"status":{"status":"write"}}},{"id":"1_1613037252.600603_328.1","section":"bucket","name":"erivkc-39","timestamp":"2021-02-11T09:54:12.600603841Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bbyzfmKm0nicNsPlFHc7TaY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037252.966061_329.1","section":"bucket.instance","name":"erivkc-40:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.40","timestamp":"2021-02-11T09:54:12.966061107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ut4zUpSzQ_IPfFkmOpHBPRo","ver":1},"status":{"status":"write"}}},{"id":"1_1613037252.971049_330.1","section":"bucket.instance","name":"erivkc-40:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.40","timestamp":"2021-02-11T09:54:12.971049650Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ut4zUpSzQ_IPfFkmOpHBPRo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037252.972980_331.1","section":"bucket","name":"erivkc-40","timestamp":"2021-02-11T09:54:12.972980368Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FZ8h3POIHnXIXFvTVd54kUz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037252.978457_332.1","section":"bucket","name":"erivkc-40","timestamp":"2021-02-11T09:54:12.978457985Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FZ8h3POIHnXIXFvTVd54kUz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037253.325406_333.1","section":"bucket.instance","name":"erivkc-41:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.41","timestamp":"2021-02-11T09:54:13.325406285Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l9qP0OWQ5TNNBXmWZGHVzV9","ver":1},"status":{"status":"write"}}},{"id":"1_1613037253.330538_334.1","section":"bucket.instance","name":"erivkc-41:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.41","timestamp":"2021-02-11T09:54:13.330538990Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l9qP0OWQ5TNNBXmWZGHVzV9","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037253.332745_335.1","section":"bucket","name":"erivkc-41","timestamp":"2021-02-11T09:54:13.332745334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hxjfNrnDah4c4Fy16F_1ybu","ver":1},"status":{"status":"write"}}},{"id":"1_1613037253.337510_336.1","section":"bucket","name":"erivkc-41","timestamp":"2021-02-11T09:54:13.337510904Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hxjfNrnDah4c4Fy16F_1ybu","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:56:32.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f2716509-104d-423b-ba5c-db1bd74642ab', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:32.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f2716509-104d-423b-ba5c-db1bd74642ab', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:32.517 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f2716509-104d-423b-ba5c-db1bd74642ab --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:32.555 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:32.555 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:32.810 INFO:teuthology.orchestra.run.smithi133.stdout:[{"id":"1_1613036845.801758_43.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T09:47:25.801758607Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lbhm83ChfGn6EHKoaQzp5TB","ver":1},"status":{"status":"write"}}},{"id":"1_1613036845.807024_48.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T09:47:25.807024959Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lbhm83ChfGn6EHKoaQzp5TB","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036964.781099_55.1","section":"bucket.instance","name":"erivkc-1:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.1","timestamp":"2021-02-11T09:49:24.781099770Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7CX9vbVJ-GdTYgNGNLhgoE1","ver":1},"status":{"status":"write"}}},{"id":"1_1613036964.794012_57.1","section":"bucket.instance","name":"erivkc-1:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.1","timestamp":"2021-02-11T09:49:24.794012217Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7CX9vbVJ-GdTYgNGNLhgoE1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036964.798399_58.1","section":"bucket","name":"erivkc-1","timestamp":"2021-02-11T09:49:24.798399196Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CPPkcG3v2weIBmWYso1zTaV","ver":1},"status":{"status":"write"}}},{"id":"1_1613036964.804170_59.1","section":"bucket","name":"erivkc-1","timestamp":"2021-02-11T09:49:24.804170325Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CPPkcG3v2weIBmWYso1zTaV","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036964.833958_60.1","section":"bucket.instance","name":"erivkc-2:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.2","timestamp":"2021-02-11T09:49:24.833958584Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2FGD8auz1ZhEdsT6v2xSbGp","ver":1},"status":{"status":"write"}}},{"id":"1_1613036964.841848_61.1","section":"bucket.instance","name":"erivkc-2:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.2","timestamp":"2021-02-11T09:49:24.841848223Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2FGD8auz1ZhEdsT6v2xSbGp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036964.843748_62.1","section":"bucket","name":"erivkc-2","timestamp":"2021-02-11T09:49:24.843748711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N0rMsI_g5Yis-zEde5_K0Ir","ver":1},"status":{"status":"write"}}},{"id":"1_1613036964.850242_63.1","section":"bucket","name":"erivkc-2","timestamp":"2021-02-11T09:49:24.850242744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N0rMsI_g5Yis-zEde5_K0Ir","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036968.651442_64.1","section":"bucket.instance","name":"erivkc-3:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.3","timestamp":"2021-02-11T09:49:28.651442746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1S35tybN40BRSEaKI2l3DIk","ver":1},"status":{"status":"write"}}},{"id":"1_1613036968.660175_65.1","section":"bucket.instance","name":"erivkc-3:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.3","timestamp":"2021-02-11T09:49:28.660175151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1S35tybN40BRSEaKI2l3DIk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036968.662383_66.1","section":"bucket","name":"erivkc-3","timestamp":"2021-02-11T09:49:28.662383146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aUbSv47O20-wQgZ_f_BQyws","ver":1},"status":{"status":"write"}}},{"id":"1_1613036968.672641_67.1","section":"bucket","name":"erivkc-3","timestamp":"2021-02-11T09:49:28.672641364Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aUbSv47O20-wQgZ_f_BQyws","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036972.560421_68.1","section":"bucket.instance","name":"erivkc-4:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.4","timestamp":"2021-02-11T09:49:32.560421693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PBJ_0UMaWtCXBaplvwFqFoX","ver":1},"status":{"status":"write"}}},{"id":"1_1613036972.565575_70.1","section":"bucket.instance","name":"erivkc-4:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.4","timestamp":"2021-02-11T09:49:32.565575234Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PBJ_0UMaWtCXBaplvwFqFoX","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036972.567393_71.1","section":"bucket","name":"erivkc-4","timestamp":"2021-02-11T09:49:32.567393622Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1FZcpPqSVgDrFiCSVeHkB-p","ver":1},"status":{"status":"write"}}},{"id":"1_1613036972.572741_72.1","section":"bucket","name":"erivkc-4","timestamp":"2021-02-11T09:49:32.572741163Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1FZcpPqSVgDrFiCSVeHkB-p","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036972.892360_73.1","section":"bucket.instance","name":"erivkc-5:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.5","timestamp":"2021-02-11T09:49:32.892360275Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_97AcbvI_qrWCvEAT4gZODL3","ver":1},"status":{"status":"write"}}},{"id":"1_1613036972.897238_74.1","section":"bucket.instance","name":"erivkc-5:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.5","timestamp":"2021-02-11T09:49:32.897238243Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_97AcbvI_qrWCvEAT4gZODL3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036972.899057_75.1","section":"bucket","name":"erivkc-5","timestamp":"2021-02-11T09:49:32.899057233Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j8qWxRQVbGg7UOusTEwf7B9","ver":1},"status":{"status":"write"}}},{"id":"1_1613036972.904781_76.1","section":"bucket","name":"erivkc-5","timestamp":"2021-02-11T09:49:32.904781579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j8qWxRQVbGg7UOusTEwf7B9","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036973.578464_77.1","section":"bucket.instance","name":"erivkc-6:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.6","timestamp":"2021-02-11T09:49:33.578464448Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y3I0bk2WnnKmeCxS7P_RG4F","ver":1},"status":{"status":"write"}}},{"id":"1_1613036973.583582_78.1","section":"bucket.instance","name":"erivkc-6:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.6","timestamp":"2021-02-11T09:49:33.583582832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y3I0bk2WnnKmeCxS7P_RG4F","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036973.585257_79.1","section":"bucket","name":"erivkc-6","timestamp":"2021-02-11T09:49:33.585257871Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a53snF3YGjGzzmA--K1O94L","ver":1},"status":{"status":"write"}}},{"id":"1_1613036973.589655_80.1","section":"bucket","name":"erivkc-6","timestamp":"2021-02-11T09:49:33.589655678Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a53snF3YGjGzzmA--K1O94L","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.632140_81.1","section":"bucket.instance","name":"erivkc-7:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.7","timestamp":"2021-02-11T09:49:41.632140668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KTbG7jYEk3iEpbVAS1YRyaR","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.637460_84.1","section":"bucket.instance","name":"erivkc-7:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.7","timestamp":"2021-02-11T09:49:41.637460231Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KTbG7jYEk3iEpbVAS1YRyaR","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.640349_85.1","section":"bucket","name":"erivkc-7","timestamp":"2021-02-11T09:49:41.640349109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.647069_86.1","section":"bucket","name":"erivkc-7","timestamp":"2021-02-11T09:49:41.647069461Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.667580_87.1","section":"bucket.instance","name":"erivkc-8:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.8","timestamp":"2021-02-11T09:49:41.667580530Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dg-o8rJKnYjuar4EHq-FKMN","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.675956_88.1","section":"bucket.instance","name":"erivkc-8:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.8","timestamp":"2021-02-11T09:49:41.675956821Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dg-o8rJKnYjuar4EHq-FKMN","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.678555_89.1","section":"bucket","name":"erivkc-8","timestamp":"2021-02-11T09:49:41.678555790Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.684839_90.1","section":"bucket","name":"erivkc-8","timestamp":"2021-02-11T09:49:41.684839859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.745183_91.1","section":"bucket.instance","name":"erivkc-9:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.9","timestamp":"2021-02-11T09:49:41.745183419Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W5J5xaWTI0Gmgb3HWRJ8MbG","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.749950_92.1","section":"bucket.instance","name":"erivkc-9:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.9","timestamp":"2021-02-11T09:49:41.749950640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W5J5xaWTI0Gmgb3HWRJ8MbG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036981.751718_93.1","section":"bucket","name":"erivkc-9","timestamp":"2021-02-11T09:49:41.751718057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":1},"status":{"status":"write"}}},{"id":"1_1613036981.757312_94.1","section":"bucket","name":"erivkc-9","timestamp":"2021-02-11T09:49:41.757312185Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036984.846663_95.1","section":"bucket","name":"erivkc-7","timestamp":"2021-02-11T09:49:44.846663475Z","data":{"read_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":1},"write_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036984.853017_97.1","section":"bucket","name":"erivkc-7","timestamp":"2021-02-11T09:49:44.853017326Z","data":{"read_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":1},"write_version":{"tag":"_CBb50G6VCQv0En5EMGWpaNk","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036984.929328_98.1","section":"bucket","name":"erivkc-8","timestamp":"2021-02-11T09:49:44.929328641Z","data":{"read_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":1},"write_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036984.936781_99.1","section":"bucket","name":"erivkc-8","timestamp":"2021-02-11T09:49:44.936781341Z","data":{"read_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":1},"write_version":{"tag":"_v7S5vxeuWFbXMQo7Zdpi6pS","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036985.064812_100.1","section":"bucket","name":"erivkc-9","timestamp":"2021-02-11T09:49:45.064812325Z","data":{"read_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":1},"write_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036985.070217_101.1","section":"bucket","name":"erivkc-9","timestamp":"2021-02-11T09:49:45.070217328Z","data":{"read_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":1},"write_version":{"tag":"_DWHTR05pCogpBIztabZ2hJY","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036999.845976_50.1","section":"bucket.instance","name":"erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10","timestamp":"2021-02-11T09:49:59.845976867Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X61poZNvtY7wornc-IkI9Ue","ver":1},"status":{"status":"write"}}},{"id":"1_1613036999.851424_52.1","section":"bucket.instance","name":"erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10","timestamp":"2021-02-11T09:49:59.851424369Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X61poZNvtY7wornc-IkI9Ue","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036999.853338_53.1","section":"bucket","name":"erivkc-10","timestamp":"2021-02-11T09:49:59.853338236Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f9x9v20LyN2Z56R02tSPZg7","ver":1},"status":{"status":"write"}}},{"id":"1_1613036999.858076_54.1","section":"bucket","name":"erivkc-10","timestamp":"2021-02-11T09:49:59.858076108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f9x9v20LyN2Z56R02tSPZg7","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036999.993526_56.1","section":"bucket.instance","name":"erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11","timestamp":"2021-02-11T09:49:59.993526854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c9r9KX8B4gD8WQmfwj_PhZZ","ver":1},"status":{"status":"write"}}},{"id":"1_1613036999.999923_57.1","section":"bucket.instance","name":"erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11","timestamp":"2021-02-11T09:49:59.999923814Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c9r9KX8B4gD8WQmfwj_PhZZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037000.001777_58.1","section":"bucket","name":"erivkc-11","timestamp":"2021-02-11T09:50:00.001777802Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dqr2cFllRHl99RGY2i9FuEy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037000.007381_59.1","section":"bucket","name":"erivkc-11","timestamp":"2021-02-11T09:50:00.007381054Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dqr2cFllRHl99RGY2i9FuEy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037000.051806_60.1","section":"bucket.instance","name":"erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12","timestamp":"2021-02-11T09:50:00.051806024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D8OCKLAmYRpy1715HaEnUrg","ver":1},"status":{"status":"write"}}},{"id":"1_1613037000.057515_61.1","section":"bucket.instance","name":"erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12","timestamp":"2021-02-11T09:50:00.057515941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D8OCKLAmYRpy1715HaEnUrg","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037000.061137_62.1","section":"bucket","name":"erivkc-12","timestamp":"2021-02-11T09:50:00.061137438Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AkdgcL9F7rK58axzIzhY-5R","ver":1},"status":{"status":"write"}}},{"id":"1_1613037000.074230_63.1","section":"bucket","name":"erivkc-12","timestamp":"2021-02-11T09:50:00.074230334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AkdgcL9F7rK58axzIzhY-5R","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.704912_167.1","section":"bucket.instance","name":"erivkc-13:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.13","timestamp":"2021-02-11T09:50:13.704912580Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wtf4rO_4_UFnmTOV6Ka-w-5","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.710270_168.1","section":"bucket.instance","name":"erivkc-13:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.13","timestamp":"2021-02-11T09:50:13.710270887Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wtf4rO_4_UFnmTOV6Ka-w-5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.712243_169.1","section":"bucket","name":"erivkc-13","timestamp":"2021-02-11T09:50:13.712243833Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xIInLmSguo5ojG9rkSrh6NV","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.717978_170.1","section":"bucket","name":"erivkc-13","timestamp":"2021-02-11T09:50:13.717978080Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xIInLmSguo5ojG9rkSrh6NV","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.741489_171.1","section":"bucket.instance","name":"erivkc-14:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.14","timestamp":"2021-02-11T09:50:13.741489385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qjqHLxJu0bVGtSl_DLxSzVc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.748782_172.1","section":"bucket.instance","name":"erivkc-14:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.14","timestamp":"2021-02-11T09:50:13.748782975Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qjqHLxJu0bVGtSl_DLxSzVc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.750682_173.1","section":"bucket","name":"erivkc-14","timestamp":"2021-02-11T09:50:13.750682310Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w3Y_zBpyOfitYjQtUFfUTuw","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.755278_174.1","section":"bucket","name":"erivkc-14","timestamp":"2021-02-11T09:50:13.755278974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w3Y_zBpyOfitYjQtUFfUTuw","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.798885_175.1","section":"bucket.instance","name":"erivkc-15:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.15","timestamp":"2021-02-11T09:50:13.798885547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8XLPKECf7E8u3tos2GxeuTy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.804452_176.1","section":"bucket.instance","name":"erivkc-15:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.15","timestamp":"2021-02-11T09:50:13.804452901Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8XLPKECf7E8u3tos2GxeuTy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037013.806359_177.1","section":"bucket","name":"erivkc-15","timestamp":"2021-02-11T09:50:13.806359983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9Prt9QKeINJuoGyCp0FIuyH","ver":1},"status":{"status":"write"}}},{"id":"1_1613037013.812144_178.1","section":"bucket","name":"erivkc-15","timestamp":"2021-02-11T09:50:13.812144543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9Prt9QKeINJuoGyCp0FIuyH","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037041.978668_183.1","section":"bucket.instance","name":"erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16","timestamp":"2021-02-11T09:50:41.978668726Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":1},"status":{"status":"write"}}},{"id":"1_1613037041.984135_184.1","section":"bucket.instance","name":"erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16","timestamp":"2021-02-11T09:50:41.984135248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037041.986045_185.1","section":"bucket","name":"erivkc-16","timestamp":"2021-02-11T09:50:41.986045547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RZKX2QLjfnqRLbyJ9ygdT-6","ver":1},"status":{"status":"write"}}},{"id":"1_1613037041.991151_186.1","section":"bucket","name":"erivkc-16","timestamp":"2021-02-11T09:50:41.991151568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RZKX2QLjfnqRLbyJ9ygdT-6","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037042.691423_187.1","section":"bucket.instance","name":"erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17","timestamp":"2021-02-11T09:50:42.691423169Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":1},"status":{"status":"write"}}},{"id":"1_1613037042.696754_188.1","section":"bucket.instance","name":"erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17","timestamp":"2021-02-11T09:50:42.696754644Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037042.698816_189.1","section":"bucket","name":"erivkc-17","timestamp":"2021-02-11T09:50:42.698816857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z7VmmfI938ifTeJ9aE0V1Ml","ver":1},"status":{"status":"write"}}},{"id":"1_1613037042.703975_190.1","section":"bucket","name":"erivkc-17","timestamp":"2021-02-11T09:50:42.703975143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z7VmmfI938ifTeJ9aE0V1Ml","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037043.653339_191.1","section":"bucket.instance","name":"erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18","timestamp":"2021-02-11T09:50:43.653339978Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":1},"status":{"status":"write"}}},{"id":"1_1613037043.659257_192.1","section":"bucket.instance","name":"erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18","timestamp":"2021-02-11T09:50:43.659257779Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037043.661142_193.1","section":"bucket","name":"erivkc-18","timestamp":"2021-02-11T09:50:43.661142206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GjOszKvzPokzjfI2YAb1ciu","ver":1},"status":{"status":"write"}}},{"id":"1_1613037043.666578_194.1","section":"bucket","name":"erivkc-18","timestamp":"2021-02-11T09:50:43.666578075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GjOszKvzPokzjfI2YAb1ciu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037044.503349_195.1","section":"bucket.instance","name":"erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16","timestamp":"2021-02-11T09:50:44.503349589Z","data":{"read_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":1},"write_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":2},"status":{"status":"write"}}},{"id":"1_1613037044.514489_196.1","section":"bucket.instance","name":"erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16","timestamp":"2021-02-11T09:50:44.514489077Z","data":{"read_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":1},"write_version":{"tag":"_Qj97yjx9M7fVTvK5h0dHQiI","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037045.153847_197.1","section":"bucket.instance","name":"erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17","timestamp":"2021-02-11T09:50:45.153847525Z","data":{"read_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":1},"write_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":2},"status":{"status":"write"}}},{"id":"1_1613037045.165645_198.1","section":"bucket.instance","name":"erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17","timestamp":"2021-02-11T09:50:45.165645371Z","data":{"read_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":1},"write_version":{"tag":"_dNaW_J-q3Kepb9aRBy4Bht3","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037045.642925_199.1","section":"bucket.instance","name":"erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18","timestamp":"2021-02-11T09:50:45.642925401Z","data":{"read_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":1},"write_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":2},"status":{"status":"write"}}},{"id":"1_1613037045.651083_200.1","section":"bucket.instance","name":"erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18","timestamp":"2021-02-11T09:50:45.651083022Z","data":{"read_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":1},"write_version":{"tag":"_WfERmhU-mJOGEaIc5_A54jk","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037125.481929_201.1","section":"bucket.instance","name":"erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19","timestamp":"2021-02-11T09:52:05.481929899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.487401_202.1","section":"bucket.instance","name":"erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19","timestamp":"2021-02-11T09:52:05.487401571Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.489373_203.1","section":"bucket","name":"erivkc-19","timestamp":"2021-02-11T09:52:05.489373157Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Jff8SGfEPMevwcwi_i-z2vJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.498307_204.1","section":"bucket","name":"erivkc-19","timestamp":"2021-02-11T09:52:05.498307854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Jff8SGfEPMevwcwi_i-z2vJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.505367_205.1","section":"bucket.instance","name":"erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19","timestamp":"2021-02-11T09:52:05.505367585Z","data":{"read_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":1},"write_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":2},"status":{"status":"write"}}},{"id":"1_1613037125.514387_206.1","section":"bucket.instance","name":"erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19","timestamp":"2021-02-11T09:52:05.514387451Z","data":{"read_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":1},"write_version":{"tag":"_nCWjYxWqNjWKl0jecb4BfEU","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037186.354909_207.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:06.354909614Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.360229_208.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:06.360229344Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037186.362241_209.1","section":"bucket","name":"erivkc-20","timestamp":"2021-02-11T09:53:06.362241066Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f2qv734GBzTTVNMfY72MNwy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.368386_210.1","section":"bucket","name":"erivkc-20","timestamp":"2021-02-11T09:53:06.368386631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f2qv734GBzTTVNMfY72MNwy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037193.931707_211.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:13.931707918Z","data":{"read_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":1},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":2},"status":{"status":"write"}}},{"id":"1_1613037193.942339_212.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:13.942339208Z","data":{"read_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":1},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037200.002098_213.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:20.002098607Z","data":{"read_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":2},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":3},"status":{"status":"write"}}},{"id":"1_1613037200.012526_214.1","section":"bucket.instance","name":"erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20","timestamp":"2021-02-11T09:53:20.012526463Z","data":{"read_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":2},"write_version":{"tag":"_aj40AZFCUvPFszWrsQyHoc0","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037207.282029_215.1","section":"bucket.instance","name":"erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21","timestamp":"2021-02-11T09:53:27.282029137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.288219_216.1","section":"bucket.instance","name":"erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21","timestamp":"2021-02-11T09:53:27.288219184Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.290439_217.1","section":"bucket","name":"erivkc-21","timestamp":"2021-02-11T09:53:27.290439853Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tvXoIwFs9HpMEOc0iK5QT1h","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.296030_218.1","section":"bucket","name":"erivkc-21","timestamp":"2021-02-11T09:53:27.296030630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tvXoIwFs9HpMEOc0iK5QT1h","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.320765_219.1","section":"bucket.instance","name":"erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22","timestamp":"2021-02-11T09:53:27.320765881Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.326634_220.1","section":"bucket.instance","name":"erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22","timestamp":"2021-02-11T09:53:27.326634616Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.328915_221.1","section":"bucket","name":"erivkc-22","timestamp":"2021-02-11T09:53:27.328915473Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zb993XEJ8BiM8mBe3hvWtO0","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.334920_222.1","section":"bucket","name":"erivkc-22","timestamp":"2021-02-11T09:53:27.334920903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zb993XEJ8BiM8mBe3hvWtO0","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.394575_223.1","section":"bucket.instance","name":"erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23","timestamp":"2021-02-11T09:53:27.394575034Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.403410_224.1","section":"bucket.instance","name":"erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23","timestamp":"2021-02-11T09:53:27.403410972Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.408075_225.1","section":"bucket","name":"erivkc-23","timestamp":"2021-02-11T09:53:27.408075932Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x_zF6jY24d41dOixQ9zJGrE","ver":1},"status":{"status":"write"}}},{"id":"1_1613037207.414692_226.1","section":"bucket","name":"erivkc-23","timestamp":"2021-02-11T09:53:27.414692449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x_zF6jY24d41dOixQ9zJGrE","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037207.445616_227.1","section":"bucket.instance","name":"erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21","timestamp":"2021-02-11T09:53:27.445616320Z","data":{"read_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":1},"write_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":2},"status":{"status":"write"}}},{"id":"1_1613037207.454205_228.1","section":"bucket.instance","name":"erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21","timestamp":"2021-02-11T09:53:27.454205301Z","data":{"read_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":1},"write_version":{"tag":"_LcBXgD3GAtZRfct3hI-MN37","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037207.463414_229.1","section":"bucket.instance","name":"erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22","timestamp":"2021-02-11T09:53:27.463414545Z","data":{"read_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":1},"write_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":2},"status":{"status":"write"}}},{"id":"1_1613037207.472168_230.1","section":"bucket.instance","name":"erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22","timestamp":"2021-02-11T09:53:27.472168737Z","data":{"read_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":1},"write_version":{"tag":"_N8yRg0VMtWBgmyUp8knMtoZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037207.485112_231.1","section":"bucket.instance","name":"erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23","timestamp":"2021-02-11T09:53:27.485112100Z","data":{"read_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":1},"write_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":2},"status":{"status":"write"}}},{"id":"1_1613037207.496661_232.1","section":"bucket.instance","name":"erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23","timestamp":"2021-02-11T09:53:27.496661758Z","data":{"read_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":1},"write_version":{"tag":"_2IhZDc7zCzQH7WR6K-NwwCy","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037225.179784_233.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.179784590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.187474_234.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.187474262Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.189621_235.1","section":"bucket","name":"erivkc-24","timestamp":"2021-02-11T09:53:45.189621882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KKbtq-SaZp1BbP3NTY0W1Qw","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.196093_236.1","section":"bucket","name":"erivkc-24","timestamp":"2021-02-11T09:53:45.196093466Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KKbtq-SaZp1BbP3NTY0W1Qw","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.222139_237.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.222139168Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.230640_238.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.230640700Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.232853_239.1","section":"bucket","name":"erivkc-25","timestamp":"2021-02-11T09:53:45.232853969Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v3PBdFYuFlkcRuNZcqiRP0s","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.239328_240.1","section":"bucket","name":"erivkc-25","timestamp":"2021-02-11T09:53:45.239328358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v3PBdFYuFlkcRuNZcqiRP0s","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.288768_241.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.288768050Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.296414_242.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.296414547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.298349_243.1","section":"bucket","name":"erivkc-26","timestamp":"2021-02-11T09:53:45.298349270Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3HLmRlz5MeuRgXHTkAYCO5-","ver":1},"status":{"status":"write"}}},{"id":"1_1613037225.307308_244.1","section":"bucket","name":"erivkc-26","timestamp":"2021-02-11T09:53:45.307308476Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3HLmRlz5MeuRgXHTkAYCO5-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037225.338581_245.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.338581131Z","data":{"read_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":1},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":2},"status":{"status":"write"}}},{"id":"1_1613037225.345749_246.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.345749522Z","data":{"read_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":1},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037225.350555_247.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.350555471Z","data":{"read_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":2},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":3},"status":{"status":"write"}}},{"id":"1_1613037225.361606_248.1","section":"bucket.instance","name":"erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24","timestamp":"2021-02-11T09:53:45.361606024Z","data":{"read_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":2},"write_version":{"tag":"_FZQlWadNKol5zSS0Xxe1zYe","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037225.372379_249.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.372379079Z","data":{"read_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":1},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":2},"status":{"status":"write"}}},{"id":"1_1613037225.386914_250.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.386914935Z","data":{"read_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":1},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037225.397898_251.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.397898539Z","data":{"read_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":2},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":3},"status":{"status":"write"}}},{"id":"1_1613037225.403904_252.1","section":"bucket.instance","name":"erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25","timestamp":"2021-02-11T09:53:45.403904049Z","data":{"read_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":2},"write_version":{"tag":"_J_bKZ_w82NUq0AWxhVnSoXp","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037225.419818_253.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.419818566Z","data":{"read_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":1},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":2},"status":{"status":"write"}}},{"id":"1_1613037225.424682_254.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.424682186Z","data":{"read_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":1},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037225.434889_255.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.434889809Z","data":{"read_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":2},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":3},"status":{"status":"write"}}},{"id":"1_1613037225.441045_256.1","section":"bucket.instance","name":"erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26","timestamp":"2021-02-11T09:53:45.441045732Z","data":{"read_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":2},"write_version":{"tag":"_nhbUAf2GjY4UllfeugrWvuu","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037242.662969_257.1","section":"bucket.instance","name":"erivkc-27:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.27","timestamp":"2021-02-11T09:54:02.662969410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.668739_258.1","section":"bucket.instance","name":"erivkc-27:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.27","timestamp":"2021-02-11T09:54:02.668739755Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.670752_259.1","section":"bucket","name":"erivkc-27","timestamp":"2021-02-11T09:54:02.670752769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gH_oydOqixBGu2q9mVl-QS_","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.675376_260.1","section":"bucket","name":"erivkc-27","timestamp":"2021-02-11T09:54:02.675376669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gH_oydOqixBGu2q9mVl-QS_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.700651_261.1","section":"bucket.instance","name":"erivkc-28:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.28","timestamp":"2021-02-11T09:54:02.700651797Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.706443_262.1","section":"bucket.instance","name":"erivkc-28:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.28","timestamp":"2021-02-11T09:54:02.706443830Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.708331_263.1","section":"bucket","name":"erivkc-28","timestamp":"2021-02-11T09:54:02.708331319Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTuUd6ahhMShO2l9OhA78Xc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.713018_264.1","section":"bucket","name":"erivkc-28","timestamp":"2021-02-11T09:54:02.713018965Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTuUd6ahhMShO2l9OhA78Xc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.762579_265.1","section":"bucket.instance","name":"erivkc-29:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.29","timestamp":"2021-02-11T09:54:02.762579200Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.775474_266.1","section":"bucket.instance","name":"erivkc-29:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.29","timestamp":"2021-02-11T09:54:02.775474031Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.782163_267.1","section":"bucket","name":"erivkc-29","timestamp":"2021-02-11T09:54:02.782163490Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OcZh0JgP4psaTeCxR4eUFzB","ver":1},"status":{"status":"write"}}},{"id":"1_1613037242.789419_268.1","section":"bucket","name":"erivkc-29","timestamp":"2021-02-11T09:54:02.789419763Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OcZh0JgP4psaTeCxR4eUFzB","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037242.824967_270.1","section":"bucket.instance","name":"erivkc-27:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.27","timestamp":"2021-02-11T09:54:02.824967755Z","data":{"read_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":1},"write_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":2},"status":{"status":"write"}}},{"id":"1_1613037242.831294_271.1","section":"bucket.instance","name":"erivkc-27:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.27","timestamp":"2021-02-11T09:54:02.831294730Z","data":{"read_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":1},"write_version":{"tag":"_6DCQprM3gwBGEhtEH7Uc3hX","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037242.879129_272.1","section":"bucket.instance","name":"erivkc-28:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.28","timestamp":"2021-02-11T09:54:02.879129288Z","data":{"read_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":1},"write_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":2},"status":{"status":"write"}}},{"id":"1_1613037242.888573_273.1","section":"bucket.instance","name":"erivkc-28:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.28","timestamp":"2021-02-11T09:54:02.888573312Z","data":{"read_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":1},"write_version":{"tag":"_7OAWE0oFJ2nqoOB-Tl2Xz_I","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037242.941315_274.1","section":"bucket.instance","name":"erivkc-29:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.29","timestamp":"2021-02-11T09:54:02.941315096Z","data":{"read_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":1},"write_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":2},"status":{"status":"write"}}},{"id":"1_1613037242.953261_275.1","section":"bucket.instance","name":"erivkc-29:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.29","timestamp":"2021-02-11T09:54:02.953261205Z","data":{"read_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":1},"write_version":{"tag":"_i34jKfCvirjAvp0gYz66MLr","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037243.018597_277.1","section":"bucket.instance","name":"erivkc-30:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.30","timestamp":"2021-02-11T09:54:03.018597532Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.024482_278.1","section":"bucket.instance","name":"erivkc-30:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.30","timestamp":"2021-02-11T09:54:03.024482726Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.026590_279.1","section":"bucket","name":"erivkc-30","timestamp":"2021-02-11T09:54:03.026590985Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b8xacbbSicSxMqmYhFg2pvb","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.032758_280.1","section":"bucket","name":"erivkc-30","timestamp":"2021-02-11T09:54:03.032758243Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b8xacbbSicSxMqmYhFg2pvb","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.060231_281.1","section":"bucket.instance","name":"erivkc-31:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.31","timestamp":"2021-02-11T09:54:03.060231918Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.066319_282.1","section":"bucket.instance","name":"erivkc-31:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.31","timestamp":"2021-02-11T09:54:03.066319997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.068298_283.1","section":"bucket","name":"erivkc-31","timestamp":"2021-02-11T09:54:03.068298381Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DRK-e9sS3ydFf_VBlB5b3Uc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.072941_284.1","section":"bucket","name":"erivkc-31","timestamp":"2021-02-11T09:54:03.072941983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DRK-e9sS3ydFf_VBlB5b3Uc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.112419_285.1","section":"bucket.instance","name":"erivkc-32:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.32","timestamp":"2021-02-11T09:54:03.112419548Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.119240_286.1","section":"bucket.instance","name":"erivkc-32:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.32","timestamp":"2021-02-11T09:54:03.119240251Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.121798_287.1","section":"bucket","name":"erivkc-32","timestamp":"2021-02-11T09:54:03.121798440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_axnp7YglcFxPJLhOhhHHGZS","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.128266_288.1","section":"bucket","name":"erivkc-32","timestamp":"2021-02-11T09:54:03.128266708Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_axnp7YglcFxPJLhOhhHHGZS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.157131_289.1","section":"bucket.instance","name":"erivkc-30:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.30","timestamp":"2021-02-11T09:54:03.157131853Z","data":{"read_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":1},"write_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":2},"status":{"status":"write"}}},{"id":"1_1613037243.168904_290.1","section":"bucket.instance","name":"erivkc-30:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.30","timestamp":"2021-02-11T09:54:03.168904231Z","data":{"read_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":1},"write_version":{"tag":"_AUU1dfn6FUuz6sGmlBWpMwd","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037243.227264_291.1","section":"bucket.instance","name":"erivkc-31:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.31","timestamp":"2021-02-11T09:54:03.227264607Z","data":{"read_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":1},"write_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":2},"status":{"status":"write"}}},{"id":"1_1613037243.232856_292.1","section":"bucket.instance","name":"erivkc-31:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.31","timestamp":"2021-02-11T09:54:03.232856122Z","data":{"read_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":1},"write_version":{"tag":"_LUGVn0xOZofYwVwR77c6G6T","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037243.287584_293.1","section":"bucket.instance","name":"erivkc-32:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.32","timestamp":"2021-02-11T09:54:03.287584331Z","data":{"read_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":1},"write_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":2},"status":{"status":"write"}}},{"id":"1_1613037243.297721_294.1","section":"bucket.instance","name":"erivkc-32:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.32","timestamp":"2021-02-11T09:54:03.297721301Z","data":{"read_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":1},"write_version":{"tag":"_Ru1gRIAqbyJz7ZZKSlzvfbz","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037243.370771_295.1","section":"bucket.instance","name":"erivkc-33:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.33","timestamp":"2021-02-11T09:54:03.370771432Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.379000_296.1","section":"bucket.instance","name":"erivkc-33:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.33","timestamp":"2021-02-11T09:54:03.379000661Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.391762_297.1","section":"bucket","name":"erivkc-33","timestamp":"2021-02-11T09:54:03.391762122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ADx5xUaeeeFNdTXMcdfXz-9","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.396407_298.1","section":"bucket","name":"erivkc-33","timestamp":"2021-02-11T09:54:03.396407953Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ADx5xUaeeeFNdTXMcdfXz-9","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.416163_299.1","section":"bucket.instance","name":"erivkc-34:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.34","timestamp":"2021-02-11T09:54:03.416163733Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.423119_300.1","section":"bucket.instance","name":"erivkc-34:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.34","timestamp":"2021-02-11T09:54:03.423119475Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.425169_301.1","section":"bucket","name":"erivkc-34","timestamp":"2021-02-11T09:54:03.425169760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ww9PzUwQW1bSFjXRQ62t6to","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.430515_302.1","section":"bucket","name":"erivkc-34","timestamp":"2021-02-11T09:54:03.430515925Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ww9PzUwQW1bSFjXRQ62t6to","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.472698_303.1","section":"bucket.instance","name":"erivkc-35:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.35","timestamp":"2021-02-11T09:54:03.472698603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.478541_304.1","section":"bucket.instance","name":"erivkc-35:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.35","timestamp":"2021-02-11T09:54:03.478541182Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.480470_305.1","section":"bucket","name":"erivkc-35","timestamp":"2021-02-11T09:54:03.480470578Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Oem2vERyFYbZLeBy-7DQmFi","ver":1},"status":{"status":"write"}}},{"id":"1_1613037243.485715_306.1","section":"bucket","name":"erivkc-35","timestamp":"2021-02-11T09:54:03.485715606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Oem2vERyFYbZLeBy-7DQmFi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037243.508665_307.1","section":"bucket.instance","name":"erivkc-33:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.33","timestamp":"2021-02-11T09:54:03.508665417Z","data":{"read_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":1},"write_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":2},"status":{"status":"write"}}},{"id":"1_1613037243.514139_308.1","section":"bucket.instance","name":"erivkc-33:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.33","timestamp":"2021-02-11T09:54:03.514139374Z","data":{"read_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":1},"write_version":{"tag":"_ENFbx-sY6QznCUGrOHybw9p","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037244.562809_309.1","section":"bucket.instance","name":"erivkc-34:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.34","timestamp":"2021-02-11T09:54:04.562809508Z","data":{"read_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":1},"write_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":2},"status":{"status":"write"}}},{"id":"1_1613037244.574097_310.1","section":"bucket.instance","name":"erivkc-34:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.34","timestamp":"2021-02-11T09:54:04.574097270Z","data":{"read_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":1},"write_version":{"tag":"_YXL95ZtqMOKTegS6zjpY8tz","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037246.158942_311.1","section":"bucket.instance","name":"erivkc-35:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.35","timestamp":"2021-02-11T09:54:06.158942088Z","data":{"read_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":1},"write_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":2},"status":{"status":"write"}}},{"id":"1_1613037246.169566_312.1","section":"bucket.instance","name":"erivkc-35:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.35","timestamp":"2021-02-11T09:54:06.169566208Z","data":{"read_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":1},"write_version":{"tag":"_uwuTQ4q-jwSZVFQw2mcAIFV","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037246.707056_313.1","section":"bucket.instance","name":"erivkc-36:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.36","timestamp":"2021-02-11T09:54:06.707056801Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xkVFLPDDAFPFhXQj_hUQN_h","ver":1},"status":{"status":"write"}}},{"id":"1_1613037246.712355_314.1","section":"bucket.instance","name":"erivkc-36:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.36","timestamp":"2021-02-11T09:54:06.712355636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xkVFLPDDAFPFhXQj_hUQN_h","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037246.714533_315.1","section":"bucket","name":"erivkc-36","timestamp":"2021-02-11T09:54:06.714533140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RHYMrzG1L0pGiaLoVMFN6Jh","ver":1},"status":{"status":"write"}}},{"id":"1_1613037246.720239_316.1","section":"bucket","name":"erivkc-36","timestamp":"2021-02-11T09:54:06.720239982Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RHYMrzG1L0pGiaLoVMFN6Jh","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037247.324144_317.1","section":"bucket.instance","name":"erivkc-37:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.37","timestamp":"2021-02-11T09:54:07.324144788Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IULRi3Z7Wy8cbA2hu8UPNbn","ver":1},"status":{"status":"write"}}},{"id":"1_1613037247.329358_318.1","section":"bucket.instance","name":"erivkc-37:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.37","timestamp":"2021-02-11T09:54:07.329358872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IULRi3Z7Wy8cbA2hu8UPNbn","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037247.331412_319.1","section":"bucket","name":"erivkc-37","timestamp":"2021-02-11T09:54:07.331412921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4WsXtBgVYqJH440dvH45LjN","ver":1},"status":{"status":"write"}}},{"id":"1_1613037247.336333_320.1","section":"bucket","name":"erivkc-37","timestamp":"2021-02-11T09:54:07.336333562Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4WsXtBgVYqJH440dvH45LjN","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037248.290625_321.1","section":"bucket.instance","name":"erivkc-38:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.38","timestamp":"2021-02-11T09:54:08.290625366Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HE4UB5TesXvy1aNWgMXkzF_","ver":1},"status":{"status":"write"}}},{"id":"1_1613037248.296282_322.1","section":"bucket.instance","name":"erivkc-38:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.38","timestamp":"2021-02-11T09:54:08.296282179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HE4UB5TesXvy1aNWgMXkzF_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037248.300696_323.1","section":"bucket","name":"erivkc-38","timestamp":"2021-02-11T09:54:08.300696565Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0eGq9xqmjVJXOf9NRs_EZL1","ver":1},"status":{"status":"write"}}},{"id":"1_1613037248.306636_324.1","section":"bucket","name":"erivkc-38","timestamp":"2021-02-11T09:54:08.306636403Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0eGq9xqmjVJXOf9NRs_EZL1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037252.586120_325.1","section":"bucket.instance","name":"erivkc-39:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.39","timestamp":"2021-02-11T09:54:12.586120488Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YrkZR2UOm5zRnAz70C8q0Ts","ver":1},"status":{"status":"write"}}},{"id":"1_1613037252.592030_326.1","section":"bucket.instance","name":"erivkc-39:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.39","timestamp":"2021-02-11T09:54:12.592030725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YrkZR2UOm5zRnAz70C8q0Ts","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037252.594399_327.1","section":"bucket","name":"erivkc-39","timestamp":"2021-02-11T09:54:12.594399304Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bbyzfmKm0nicNsPlFHc7TaY","ver":1},"status":{"status":"write"}}},{"id":"1_1613037252.600603_328.1","section":"bucket","name":"erivkc-39","timestamp":"2021-02-11T09:54:12.600603841Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bbyzfmKm0nicNsPlFHc7TaY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037252.966061_329.1","section":"bucket.instance","name":"erivkc-40:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.40","timestamp":"2021-02-11T09:54:12.966061107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ut4zUpSzQ_IPfFkmOpHBPRo","ver":1},"status":{"status":"write"}}},{"id":"1_1613037252.971049_330.1","section":"bucket.instance","name":"erivkc-40:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.40","timestamp":"2021-02-11T09:54:12.971049650Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ut4zUpSzQ_IPfFkmOpHBPRo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037252.972980_331.1","section":"bucket","name":"erivkc-40","timestamp":"2021-02-11T09:54:12.972980368Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FZ8h3POIHnXIXFvTVd54kUz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037252.978457_332.1","section":"bucket","name":"erivkc-40","timestamp":"2021-02-11T09:54:12.978457985Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FZ8h3POIHnXIXFvTVd54kUz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037253.325406_333.1","section":"bucket.instance","name":"erivkc-41:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.41","timestamp":"2021-02-11T09:54:13.325406285Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l9qP0OWQ5TNNBXmWZGHVzV9","ver":1},"status":{"status":"write"}}},{"id":"1_1613037253.330538_334.1","section":"bucket.instance","name":"erivkc-41:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.41","timestamp":"2021-02-11T09:54:13.330538990Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l9qP0OWQ5TNNBXmWZGHVzV9","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037253.332745_335.1","section":"bucket","name":"erivkc-41","timestamp":"2021-02-11T09:54:13.332745334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hxjfNrnDah4c4Fy16F_1ybu","ver":1},"status":{"status":"write"}}},{"id":"1_1613037253.337510_336.1","section":"bucket","name":"erivkc-41","timestamp":"2021-02-11T09:54:13.337510904Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hxjfNrnDah4c4Fy16F_1ybu","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:56:32.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '8ead83cf-fd6c-4023-bc3d-a277ca9366d6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:32.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '8ead83cf-fd6c-4023-bc3d-a277ca9366d6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:32.812 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 8ead83cf-fd6c-4023-bc3d-a277ca9366d6 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:32.851 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:32.851 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:33.101 INFO:teuthology.orchestra.run.smithi026.stdout:[{"id":"1_1613037321.098409_343.1","section":"bucket.instance","name":"erivkc-42:2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.1","timestamp":"2021-02-11T09:55:21.098409384Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zcd383RbvBRyIHDKXw3pQN0","ver":1},"status":{"status":"write"}}},{"id":"1_1613037321.103728_344.1","section":"bucket.instance","name":"erivkc-42:2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.1","timestamp":"2021-02-11T09:55:21.103728846Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zcd383RbvBRyIHDKXw3pQN0","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037321.105592_345.1","section":"bucket","name":"erivkc-42","timestamp":"2021-02-11T09:55:21.105592564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QbRENFjjc0U3QSFu-QTnddx","ver":1},"status":{"status":"write"}}},{"id":"1_1613037321.110075_346.1","section":"bucket","name":"erivkc-42","timestamp":"2021-02-11T09:55:21.110075179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QbRENFjjc0U3QSFu-QTnddx","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037321.154882_347.1","section":"bucket.instance","name":"erivkc-43:2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.2","timestamp":"2021-02-11T09:55:21.154882325Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CXPwdqK63_YBqV-bubPr5KT","ver":1},"status":{"status":"write"}}},{"id":"1_1613037321.161689_348.1","section":"bucket.instance","name":"erivkc-43:2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.2","timestamp":"2021-02-11T09:55:21.161689366Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CXPwdqK63_YBqV-bubPr5KT","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037321.163711_349.1","section":"bucket","name":"erivkc-43","timestamp":"2021-02-11T09:55:21.163711882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9XIqNXgKEOVqEmuilv2usEp","ver":1},"status":{"status":"write"}}},{"id":"1_1613037321.168858_350.1","section":"bucket","name":"erivkc-43","timestamp":"2021-02-11T09:55:21.168858625Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9XIqNXgKEOVqEmuilv2usEp","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:56:33.102 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '8ead83cf-fd6c-4023-bc3d-a277ca9366d6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:33.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '8ead83cf-fd6c-4023-bc3d-a277ca9366d6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:33.102 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 8ead83cf-fd6c-4023-bc3d-a277ca9366d6 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:33.141 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:33.141 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:33.390 INFO:teuthology.orchestra.run.smithi133.stdout:[{"id":"1_1613037321.098409_343.1","section":"bucket.instance","name":"erivkc-42:2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.1","timestamp":"2021-02-11T09:55:21.098409384Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zcd383RbvBRyIHDKXw3pQN0","ver":1},"status":{"status":"write"}}},{"id":"1_1613037321.103728_344.1","section":"bucket.instance","name":"erivkc-42:2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.1","timestamp":"2021-02-11T09:55:21.103728846Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zcd383RbvBRyIHDKXw3pQN0","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037321.105592_345.1","section":"bucket","name":"erivkc-42","timestamp":"2021-02-11T09:55:21.105592564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QbRENFjjc0U3QSFu-QTnddx","ver":1},"status":{"status":"write"}}},{"id":"1_1613037321.110075_346.1","section":"bucket","name":"erivkc-42","timestamp":"2021-02-11T09:55:21.110075179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QbRENFjjc0U3QSFu-QTnddx","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037321.154882_347.1","section":"bucket.instance","name":"erivkc-43:2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.2","timestamp":"2021-02-11T09:55:21.154882325Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CXPwdqK63_YBqV-bubPr5KT","ver":1},"status":{"status":"write"}}},{"id":"1_1613037321.161689_348.1","section":"bucket.instance","name":"erivkc-43:2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.2","timestamp":"2021-02-11T09:55:21.161689366Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CXPwdqK63_YBqV-bubPr5KT","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037321.163711_349.1","section":"bucket","name":"erivkc-43","timestamp":"2021-02-11T09:55:21.163711882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9XIqNXgKEOVqEmuilv2usEp","ver":1},"status":{"status":"write"}}},{"id":"1_1613037321.168858_350.1","section":"bucket","name":"erivkc-43","timestamp":"2021-02-11T09:55:21.168858625Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9XIqNXgKEOVqEmuilv2usEp","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:56:33.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '8ead83cf-fd6c-4023-bc3d-a277ca9366d6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:33.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '8ead83cf-fd6c-4023-bc3d-a277ca9366d6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:33.391 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 8ead83cf-fd6c-4023-bc3d-a277ca9366d6 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:33.432 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:33.697 INFO:teuthology.orchestra.run.smithi026.stdout:[{"id":"1_1613037321.098409_343.1","section":"bucket.instance","name":"erivkc-42:2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.1","timestamp":"2021-02-11T09:55:21.098409384Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zcd383RbvBRyIHDKXw3pQN0","ver":1},"status":{"status":"write"}}},{"id":"1_1613037321.103728_344.1","section":"bucket.instance","name":"erivkc-42:2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.1","timestamp":"2021-02-11T09:55:21.103728846Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zcd383RbvBRyIHDKXw3pQN0","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037321.105592_345.1","section":"bucket","name":"erivkc-42","timestamp":"2021-02-11T09:55:21.105592564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QbRENFjjc0U3QSFu-QTnddx","ver":1},"status":{"status":"write"}}},{"id":"1_1613037321.110075_346.1","section":"bucket","name":"erivkc-42","timestamp":"2021-02-11T09:55:21.110075179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QbRENFjjc0U3QSFu-QTnddx","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037321.154882_347.1","section":"bucket.instance","name":"erivkc-43:2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.2","timestamp":"2021-02-11T09:55:21.154882325Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CXPwdqK63_YBqV-bubPr5KT","ver":1},"status":{"status":"write"}}},{"id":"1_1613037321.161689_348.1","section":"bucket.instance","name":"erivkc-43:2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.2","timestamp":"2021-02-11T09:55:21.161689366Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CXPwdqK63_YBqV-bubPr5KT","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037321.163711_349.1","section":"bucket","name":"erivkc-43","timestamp":"2021-02-11T09:55:21.163711882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9XIqNXgKEOVqEmuilv2usEp","ver":1},"status":{"status":"write"}}},{"id":"1_1613037321.168858_350.1","section":"bucket","name":"erivkc-43","timestamp":"2021-02-11T09:55:21.168858625Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9XIqNXgKEOVqEmuilv2usEp","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:56:33.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '8ead83cf-fd6c-4023-bc3d-a277ca9366d6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:33.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '8ead83cf-fd6c-4023-bc3d-a277ca9366d6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:33.698 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 8ead83cf-fd6c-4023-bc3d-a277ca9366d6 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:33.735 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:33.735 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:33.975 INFO:teuthology.orchestra.run.smithi133.stdout:[{"id":"1_1613037321.098409_343.1","section":"bucket.instance","name":"erivkc-42:2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.1","timestamp":"2021-02-11T09:55:21.098409384Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zcd383RbvBRyIHDKXw3pQN0","ver":1},"status":{"status":"write"}}},{"id":"1_1613037321.103728_344.1","section":"bucket.instance","name":"erivkc-42:2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.1","timestamp":"2021-02-11T09:55:21.103728846Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zcd383RbvBRyIHDKXw3pQN0","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037321.105592_345.1","section":"bucket","name":"erivkc-42","timestamp":"2021-02-11T09:55:21.105592564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QbRENFjjc0U3QSFu-QTnddx","ver":1},"status":{"status":"write"}}},{"id":"1_1613037321.110075_346.1","section":"bucket","name":"erivkc-42","timestamp":"2021-02-11T09:55:21.110075179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QbRENFjjc0U3QSFu-QTnddx","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037321.154882_347.1","section":"bucket.instance","name":"erivkc-43:2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.2","timestamp":"2021-02-11T09:55:21.154882325Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CXPwdqK63_YBqV-bubPr5KT","ver":1},"status":{"status":"write"}}},{"id":"1_1613037321.161689_348.1","section":"bucket.instance","name":"erivkc-43:2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.2","timestamp":"2021-02-11T09:55:21.161689366Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CXPwdqK63_YBqV-bubPr5KT","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037321.163711_349.1","section":"bucket","name":"erivkc-43","timestamp":"2021-02-11T09:55:21.163711882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9XIqNXgKEOVqEmuilv2usEp","ver":1},"status":{"status":"write"}}},{"id":"1_1613037321.168858_350.1","section":"bucket","name":"erivkc-43","timestamp":"2021-02-11T09:55:21.168858625Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9XIqNXgKEOVqEmuilv2usEp","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:56:33.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f4d7bfad-1742-43a0-adcc-1b0cde4c6a67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:33.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f4d7bfad-1742-43a0-adcc-1b0cde4c6a67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:33.976 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:34.017 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:34.017 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:34.266 INFO:teuthology.orchestra.run.smithi026.stdout:[{"id":"1_1613037382.295954_148.1","section":"bucket.instance","name":"erivkc-44:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.1","timestamp":"2021-02-11T09:56:22.295954693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TaFcVms9tSE9AaVjJ9d4-HB","ver":1},"status":{"status":"write"}}},{"id":"1_1613037382.301507_149.1","section":"bucket.instance","name":"erivkc-44:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.1","timestamp":"2021-02-11T09:56:22.301507101Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TaFcVms9tSE9AaVjJ9d4-HB","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037382.303492_150.1","section":"bucket","name":"erivkc-44","timestamp":"2021-02-11T09:56:22.303492300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uHM1Ivm1-4vFar8oyjaTAbD","ver":1},"status":{"status":"write"}}},{"id":"1_1613037382.308510_151.1","section":"bucket","name":"erivkc-44","timestamp":"2021-02-11T09:56:22.308510832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uHM1Ivm1-4vFar8oyjaTAbD","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037382.350045_152.1","section":"bucket.instance","name":"erivkc-45:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.2","timestamp":"2021-02-11T09:56:22.350045036Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gX41P_VO8ZHRXHu5AzBq7uG","ver":1},"status":{"status":"write"}}},{"id":"1_1613037382.355245_153.1","section":"bucket.instance","name":"erivkc-45:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.2","timestamp":"2021-02-11T09:56:22.355245828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gX41P_VO8ZHRXHu5AzBq7uG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037382.357252_154.1","section":"bucket","name":"erivkc-45","timestamp":"2021-02-11T09:56:22.357252297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q2u3-y0oB6ofq2Mqul4AgNz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037382.362758_155.1","section":"bucket","name":"erivkc-45","timestamp":"2021-02-11T09:56:22.362758124Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q2u3-y0oB6ofq2Mqul4AgNz","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:56:34.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f4d7bfad-1742-43a0-adcc-1b0cde4c6a67', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:34.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f4d7bfad-1742-43a0-adcc-1b0cde4c6a67', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:34.267 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:34.305 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:34.305 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:34.547 INFO:teuthology.orchestra.run.smithi133.stdout:[{"id":"1_1613037382.295954_148.1","section":"bucket.instance","name":"erivkc-44:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.1","timestamp":"2021-02-11T09:56:22.295954693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TaFcVms9tSE9AaVjJ9d4-HB","ver":1},"status":{"status":"write"}}},{"id":"1_1613037382.301507_149.1","section":"bucket.instance","name":"erivkc-44:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.1","timestamp":"2021-02-11T09:56:22.301507101Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TaFcVms9tSE9AaVjJ9d4-HB","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037382.303492_150.1","section":"bucket","name":"erivkc-44","timestamp":"2021-02-11T09:56:22.303492300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uHM1Ivm1-4vFar8oyjaTAbD","ver":1},"status":{"status":"write"}}},{"id":"1_1613037382.308510_151.1","section":"bucket","name":"erivkc-44","timestamp":"2021-02-11T09:56:22.308510832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uHM1Ivm1-4vFar8oyjaTAbD","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037382.350045_152.1","section":"bucket.instance","name":"erivkc-45:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.2","timestamp":"2021-02-11T09:56:22.350045036Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gX41P_VO8ZHRXHu5AzBq7uG","ver":1},"status":{"status":"write"}}},{"id":"1_1613037382.355245_153.1","section":"bucket.instance","name":"erivkc-45:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.2","timestamp":"2021-02-11T09:56:22.355245828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gX41P_VO8ZHRXHu5AzBq7uG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037382.357252_154.1","section":"bucket","name":"erivkc-45","timestamp":"2021-02-11T09:56:22.357252297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q2u3-y0oB6ofq2Mqul4AgNz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037382.362758_155.1","section":"bucket","name":"erivkc-45","timestamp":"2021-02-11T09:56:22.362758124Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q2u3-y0oB6ofq2Mqul4AgNz","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:56:34.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f4d7bfad-1742-43a0-adcc-1b0cde4c6a67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:34.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f4d7bfad-1742-43a0-adcc-1b0cde4c6a67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:34.548 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:34.588 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:34.589 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:34.835 INFO:teuthology.orchestra.run.smithi026.stdout:[{"id":"1_1613037382.295954_148.1","section":"bucket.instance","name":"erivkc-44:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.1","timestamp":"2021-02-11T09:56:22.295954693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TaFcVms9tSE9AaVjJ9d4-HB","ver":1},"status":{"status":"write"}}},{"id":"1_1613037382.301507_149.1","section":"bucket.instance","name":"erivkc-44:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.1","timestamp":"2021-02-11T09:56:22.301507101Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TaFcVms9tSE9AaVjJ9d4-HB","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037382.303492_150.1","section":"bucket","name":"erivkc-44","timestamp":"2021-02-11T09:56:22.303492300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uHM1Ivm1-4vFar8oyjaTAbD","ver":1},"status":{"status":"write"}}},{"id":"1_1613037382.308510_151.1","section":"bucket","name":"erivkc-44","timestamp":"2021-02-11T09:56:22.308510832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uHM1Ivm1-4vFar8oyjaTAbD","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037382.350045_152.1","section":"bucket.instance","name":"erivkc-45:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.2","timestamp":"2021-02-11T09:56:22.350045036Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gX41P_VO8ZHRXHu5AzBq7uG","ver":1},"status":{"status":"write"}}},{"id":"1_1613037382.355245_153.1","section":"bucket.instance","name":"erivkc-45:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.2","timestamp":"2021-02-11T09:56:22.355245828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gX41P_VO8ZHRXHu5AzBq7uG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037382.357252_154.1","section":"bucket","name":"erivkc-45","timestamp":"2021-02-11T09:56:22.357252297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q2u3-y0oB6ofq2Mqul4AgNz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037382.362758_155.1","section":"bucket","name":"erivkc-45","timestamp":"2021-02-11T09:56:22.362758124Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q2u3-y0oB6ofq2Mqul4AgNz","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:56:34.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f4d7bfad-1742-43a0-adcc-1b0cde4c6a67', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:34.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f4d7bfad-1742-43a0-adcc-1b0cde4c6a67', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:34.836 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:34.876 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:34.876 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:35.124 INFO:teuthology.orchestra.run.smithi133.stdout:[{"id":"1_1613037382.295954_148.1","section":"bucket.instance","name":"erivkc-44:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.1","timestamp":"2021-02-11T09:56:22.295954693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TaFcVms9tSE9AaVjJ9d4-HB","ver":1},"status":{"status":"write"}}},{"id":"1_1613037382.301507_149.1","section":"bucket.instance","name":"erivkc-44:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.1","timestamp":"2021-02-11T09:56:22.301507101Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TaFcVms9tSE9AaVjJ9d4-HB","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037382.303492_150.1","section":"bucket","name":"erivkc-44","timestamp":"2021-02-11T09:56:22.303492300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uHM1Ivm1-4vFar8oyjaTAbD","ver":1},"status":{"status":"write"}}},{"id":"1_1613037382.308510_151.1","section":"bucket","name":"erivkc-44","timestamp":"2021-02-11T09:56:22.308510832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uHM1Ivm1-4vFar8oyjaTAbD","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037382.350045_152.1","section":"bucket.instance","name":"erivkc-45:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.2","timestamp":"2021-02-11T09:56:22.350045036Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gX41P_VO8ZHRXHu5AzBq7uG","ver":1},"status":{"status":"write"}}},{"id":"1_1613037382.355245_153.1","section":"bucket.instance","name":"erivkc-45:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.2","timestamp":"2021-02-11T09:56:22.355245828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gX41P_VO8ZHRXHu5AzBq7uG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037382.357252_154.1","section":"bucket","name":"erivkc-45","timestamp":"2021-02-11T09:56:22.357252297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q2u3-y0oB6ofq2Mqul4AgNz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037382.362758_155.1","section":"bucket","name":"erivkc-45","timestamp":"2021-02-11T09:56:22.362758124Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q2u3-y0oB6ofq2Mqul4AgNz","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:56:35.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:35.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:35.125 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:56:35.163 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:35.163 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:35.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:35.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:35.484 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:56:35.522 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:35.523 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:35.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:35.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:35.923 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:56:35.958 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:35.958 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:36.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:36.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:36.231 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:56:36.267 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:36.267 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:36.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f2716509-104d-423b-ba5c-db1bd74642ab', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:36.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f2716509-104d-423b-ba5c-db1bd74642ab', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:36.606 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f2716509-104d-423b-ba5c-db1bd74642ab --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:36.644 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:36.644 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:36.910 INFO:teuthology.orchestra.run.smithi026.stdout:[] 2021-02-11T09:56:36.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f2716509-104d-423b-ba5c-db1bd74642ab', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:36.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f2716509-104d-423b-ba5c-db1bd74642ab', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:36.911 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f2716509-104d-423b-ba5c-db1bd74642ab --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:36.951 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:36.951 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:37.211 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2021-02-11T09:56:37.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f2716509-104d-423b-ba5c-db1bd74642ab', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:37.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f2716509-104d-423b-ba5c-db1bd74642ab', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:37.211 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f2716509-104d-423b-ba5c-db1bd74642ab --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:37.251 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:37.251 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:37.502 INFO:teuthology.orchestra.run.smithi026.stdout:[] 2021-02-11T09:56:37.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f2716509-104d-423b-ba5c-db1bd74642ab', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:37.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f2716509-104d-423b-ba5c-db1bd74642ab', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:37.502 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f2716509-104d-423b-ba5c-db1bd74642ab --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:37.541 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:37.542 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:37.788 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2021-02-11T09:56:37.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '8ead83cf-fd6c-4023-bc3d-a277ca9366d6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:37.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '8ead83cf-fd6c-4023-bc3d-a277ca9366d6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:37.789 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 8ead83cf-fd6c-4023-bc3d-a277ca9366d6 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:37.828 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:37.829 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:38.077 INFO:teuthology.orchestra.run.smithi026.stdout:[] 2021-02-11T09:56:38.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '8ead83cf-fd6c-4023-bc3d-a277ca9366d6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:38.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '8ead83cf-fd6c-4023-bc3d-a277ca9366d6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:38.077 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 8ead83cf-fd6c-4023-bc3d-a277ca9366d6 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:38.115 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:38.115 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:38.374 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2021-02-11T09:56:38.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '8ead83cf-fd6c-4023-bc3d-a277ca9366d6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:38.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '8ead83cf-fd6c-4023-bc3d-a277ca9366d6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:38.375 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 8ead83cf-fd6c-4023-bc3d-a277ca9366d6 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:38.412 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:38.412 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:38.676 INFO:teuthology.orchestra.run.smithi026.stdout:[] 2021-02-11T09:56:38.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '8ead83cf-fd6c-4023-bc3d-a277ca9366d6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:38.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '8ead83cf-fd6c-4023-bc3d-a277ca9366d6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:38.677 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 8ead83cf-fd6c-4023-bc3d-a277ca9366d6 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:38.719 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:38.719 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:38.987 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2021-02-11T09:56:38.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f4d7bfad-1742-43a0-adcc-1b0cde4c6a67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:38.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f4d7bfad-1742-43a0-adcc-1b0cde4c6a67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:38.988 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:39.027 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:39.027 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:39.269 INFO:teuthology.orchestra.run.smithi026.stdout:[] 2021-02-11T09:56:39.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f4d7bfad-1742-43a0-adcc-1b0cde4c6a67', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:39.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f4d7bfad-1742-43a0-adcc-1b0cde4c6a67', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:39.269 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:39.307 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:39.307 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:39.552 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2021-02-11T09:56:39.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f4d7bfad-1742-43a0-adcc-1b0cde4c6a67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:39.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f4d7bfad-1742-43a0-adcc-1b0cde4c6a67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:39.553 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:39.593 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:39.593 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:39.860 INFO:teuthology.orchestra.run.smithi026.stdout:[] 2021-02-11T09:56:39.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f4d7bfad-1742-43a0-adcc-1b0cde4c6a67', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:39.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f4d7bfad-1742-43a0-adcc-1b0cde4c6a67', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:39.861 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:39.899 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:39.899 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:40.135 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2021-02-11T09:56:40.135 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2021-02-11T09:56:40.136 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-46 2021-02-11T09:56:40.198 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-47 2021-02-11T09:56:40.970 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-48 2021-02-11T09:56:42.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:42.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:42.531 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:42.571 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:42.571 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T09:56:42.784 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613037401.998485_196.1","last_update":"2021-02-11T09:56:41.998485Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:56:42.784 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:56:42.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:42.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:42.785 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:42.824 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:42.825 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:43.045 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037401.998485_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:41.998485Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:56:43.046 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:56:43.046 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:56:43.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:43.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:43.046 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:43.087 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:43.087 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:43.295 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037401.998485_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:41.998485Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:56:43.295 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:56:43.295 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:56:43.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:43.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:43.296 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:43.337 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:43.337 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:43.582 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037401.998485_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:41.998485Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:56:43.582 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:56:43.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:43.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:43.582 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:43.621 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:43.621 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:43.866 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037403.017566_293.1","last_update":"2021-02-11T09:56:43.017566Z"},{"marker":"1_1613037144.076572_157.1","last_update":"2021-02-11T09:52:24.076572Z"},{"marker":"1_1613037251.805432_108.1","last_update":"2021-02-11T09:54:11.805432Z"},{"marker":"1_1613037100.066735_115.3","last_update":"2021-02-11T09:51:40.066735Z"}] 2021-02-11T09:56:43.867 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T09:56:43.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:43.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:43.867 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:43.910 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:43.910 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:44.158 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037403.017566_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:43.017566Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037144.076572_157.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.076572Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037251.805432_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:54:11.805432Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037100.066735_115.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:51:40.066735Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:56:44.158 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T09:56:44.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:44.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:44.159 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:44.198 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:44.198 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:44.445 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037403.017566_293.1","last_update":"2021-02-11T09:56:43.017566Z"},{"marker":"1_1613037144.076572_157.1","last_update":"2021-02-11T09:52:24.076572Z"},{"marker":"1_1613037251.805432_108.1","last_update":"2021-02-11T09:54:11.805432Z"},{"marker":"1_1613037100.066735_115.3","last_update":"2021-02-11T09:51:40.066735Z"}] 2021-02-11T09:56:44.446 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T09:56:44.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:44.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:44.446 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:44.530 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:44.530 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:44.788 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7045610624405686142},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037403.017566_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:43.017566Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037144.076572_157.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.076572Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037251.805432_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:54:11.805432Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037100.066735_115.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:51:40.066735Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:56:44.788 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T09:56:44.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:44.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:44.789 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:44.869 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:44.869 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:45.123 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037403.017566_293.1","last_update":"2021-02-11T09:56:43.017566Z"},{"marker":"1_1613037144.076572_157.1","last_update":"2021-02-11T09:52:24.076572Z"},{"marker":"1_1613037251.805432_108.1","last_update":"2021-02-11T09:54:11.805432Z"},{"marker":"1_1613037100.066735_115.3","last_update":"2021-02-11T09:51:40.066735Z"}] 2021-02-11T09:56:45.123 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T09:56:45.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:45.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:45.124 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:45.166 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:45.166 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:45.399 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1531294025241915737},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037403.017566_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:43.017566Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037144.076572_157.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.076572Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037251.805432_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:54:11.805432Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037100.066735_115.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:51:40.066735Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:56:45.399 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T09:56:45.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:45.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:45.400 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:45.480 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:45.480 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:45.707 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1613037402.960739_274.1","last_update":"2021-02-11T09:56:42.960739Z"},{"marker":"1_1613037144.042530_160.1","last_update":"2021-02-11T09:52:24.042530Z"},{"marker":"1_1613037251.801063_115.1","last_update":"2021-02-11T09:54:11.801063Z"},{"marker":"1_1613037100.031804_163.3","last_update":"2021-02-11T09:51:40.031804Z"}] 2021-02-11T09:56:45.707 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T09:56:45.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:45.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:45.708 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:45.753 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:45.753 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:45.989 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3492294689851219663},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037402.960739_274.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:42.960739Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037144.042530_160.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.042530Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037251.801063_115.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:54:11.801063Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037100.031804_163.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:51:40.031804Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:56:45.990 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T09:56:45.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:45.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:45.990 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:46.032 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:46.032 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:46.271 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1613037402.960739_274.1","last_update":"2021-02-11T09:56:42.960739Z"},{"marker":"1_1613037144.042530_160.1","last_update":"2021-02-11T09:52:24.042530Z"},{"marker":"1_1613037251.801063_115.1","last_update":"2021-02-11T09:54:11.801063Z"},{"marker":"1_1613037100.031804_163.3","last_update":"2021-02-11T09:51:40.031804Z"}] 2021-02-11T09:56:46.271 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T09:56:46.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:46.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:46.272 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:46.316 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:46.316 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:46.579 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7442380677805818242},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037402.960739_274.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:42.960739Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037144.042530_160.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.042530Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037251.801063_115.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:54:11.801063Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037100.031804_163.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:51:40.031804Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:56:46.579 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T09:56:46.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:46.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:46.580 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:46.618 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:46.618 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:46.862 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037402.590050_273.1","last_update":"2021-02-11T09:56:42.590050Z"},{"marker":"1_1613037144.107937_91.1","last_update":"2021-02-11T09:52:24.107937Z"},{"marker":"1_1613037251.529119_87.1","last_update":"2021-02-11T09:54:11.529119Z"},{"marker":"1_1613037100.096926_105.3","last_update":"2021-02-11T09:51:40.096926Z"}] 2021-02-11T09:56:46.863 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T09:56:46.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:46.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:46.863 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:46.945 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:46.945 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:47.197 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14314976040552073418},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037402.590050_273.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:42.590050Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037144.107937_91.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.107937Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037251.529119_87.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:54:11.529119Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037100.096926_105.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:51:40.096926Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:56:47.198 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T09:56:47.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:47.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:47.198 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:47.277 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:47.277 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:47.532 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037402.590050_273.1","last_update":"2021-02-11T09:56:42.590050Z"},{"marker":"1_1613037144.107937_91.1","last_update":"2021-02-11T09:52:24.107937Z"},{"marker":"1_1613037251.529119_87.1","last_update":"2021-02-11T09:54:11.529119Z"},{"marker":"1_1613037100.096926_105.3","last_update":"2021-02-11T09:51:40.096926Z"}] 2021-02-11T09:56:47.533 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T09:56:47.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:47.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:47.533 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:47.574 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:47.574 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:47.813 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12589397483957797345},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037402.590050_273.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:42.590050Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037144.107937_91.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:52:24.107937Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037251.529119_87.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:54:11.529119Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037100.096926_105.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:51:40.096926Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:56:47.814 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T09:56:47.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:47.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:47.815 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:47.852 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:47.853 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:48.111 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037403.017566_293.1","last_update":"2021-02-11T09:56:43.017566Z"},{"marker":"1_1613037144.076572_157.1","last_update":"2021-02-11T09:52:24.076572Z"},{"marker":"1_1613037251.805432_108.1","last_update":"2021-02-11T09:54:11.805432Z"},{"marker":"1_1613037100.066735_115.3","last_update":"2021-02-11T09:51:40.066735Z"}] 2021-02-11T09:56:48.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:48.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:48.111 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:56:48.186 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:48.186 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:48.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:48.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:48.457 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:48.492 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:48.493 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:48.724 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1613037402.960739_274.1","last_update":"2021-02-11T09:56:42.960739Z"},{"marker":"1_1613037144.042530_160.1","last_update":"2021-02-11T09:52:24.042530Z"},{"marker":"1_1613037251.801063_115.1","last_update":"2021-02-11T09:54:11.801063Z"},{"marker":"1_1613037100.031804_163.3","last_update":"2021-02-11T09:51:40.031804Z"}] 2021-02-11T09:56:48.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:48.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:48.725 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:56:48.800 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:48.800 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:49.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:49.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:56:49.061 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:56:49.098 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:49.098 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:49.359 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037403.017566_293.1","last_update":"2021-02-11T09:56:43.017566Z"},{"marker":"1_1613037144.076572_157.1","last_update":"2021-02-11T09:52:24.076572Z"},{"marker":"1_1613037251.805432_108.1","last_update":"2021-02-11T09:54:11.805432Z"},{"marker":"1_1613037100.066735_115.3","last_update":"2021-02-11T09:51:40.066735Z"}] 2021-02-11T09:56:49.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:49.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:49.359 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:56:49.433 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:49.433 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:49.683 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ok 2021-02-11T09:56:49.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:49.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:49.684 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:56:49.723 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:49.723 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:49.754 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:56:49.749+0000 7fb98005e080 0 failed reading obj info from .rgw.root:zone_info.30c2afa9-98cd-42de-acef-f4a612d65cea: (2) No such file or directory 2021-02-11T09:56:49.754 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:56:49.749+0000 7fb98005e080 0 WARNING: could not read zone params for zone id=30c2afa9-98cd-42de-acef-f4a612d65cea name=test-zone3 2021-02-11T09:56:49.755 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:56:49.753+0000 7fb98005e080 0 failed reading obj info from .rgw.root:zone_info.a3c4910a-4529-4294-aed4-ef409f209cf5: (2) No such file or directory 2021-02-11T09:56:49.755 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:56:49.753+0000 7fb98005e080 0 WARNING: could not read zone params for zone id=a3c4910a-4529-4294-aed4-ef409f209cf5 name=test-zone1 2021-02-11T09:56:49.768 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"DHA6OWMQ42PGF1T7","secret_key":"xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T09:56:49.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:49.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:49.768 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:56:49.851 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:49.851 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:50.082 INFO:teuthology.orchestra.run.smithi133.stderr:Sending period to new master zone a3c4910a-4529-4294-aed4-ef409f209cf5 2021-02-11T09:56:50.195 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","epoch":2,"predecessor_uuid":"8ead83cf-fd6c-4023-bc3d-a277ca9366d6","sync_status":["","1_1613037321.168858_350.1","",""],"period_map":{"id":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","zones":[{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":""},{"id":"30c2afa9-98cd-42de-acef-f4a612d65cea","name":"test-zone3","endpoints":["http://smithi026.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"a3c4910a-4529-4294-aed4-ef409f209cf5","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","val":3299024572},{"key":"30c2afa9-98cd-42de-acef-f4a612d65cea","val":2754623789},{"key":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","val":3684997273},{"key":"a3c4910a-4529-4294-aed4-ef409f209cf5","val":2693415762}]},"master_zonegroup":"test-zonegroup","master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","realm_name":"test-realm","realm_epoch":4} 2021-02-11T09:56:50.195 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2021-02-11T09:57:50.196 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-49 2021-02-11T09:57:50.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:57:50.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:57:50.295 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:57:50.334 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:57:50.335 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:57:50.566 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T09:57:50.604 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613037470.242184_221.1","last_update":"2021-02-11T09:57:50.242184Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:57:50.605 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:57:50.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:57:50.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:57:50.605 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:57:50.647 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:57:50.647 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:57:50.895 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037401.998485_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:41.998485Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:57:50.895 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037470.242184_221.1 target=1_1613037401.998485_196.1 2021-02-11T09:57:55.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:57:55.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:57:55.897 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:57:55.936 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:57:55.936 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:57:56.194 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037401.998485_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:41.998485Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:57:56.194 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037470.242184_221.1 target=1_1613037401.998485_196.1 2021-02-11T09:58:01.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:01.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:01.196 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:01.238 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:01.238 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:01.486 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037401.998485_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:41.998485Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:58:01.487 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037470.242184_221.1 target=1_1613037401.998485_196.1 2021-02-11T09:58:06.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:06.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:06.488 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:06.528 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:06.528 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:06.744 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037401.998485_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:41.998485Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:58:06.744 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037470.242184_221.1 target=1_1613037401.998485_196.1 2021-02-11T09:58:11.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:11.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:11.746 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:11.786 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:11.786 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:12.027 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037401.998485_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:41.998485Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:58:12.027 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037470.242184_221.1 target=1_1613037401.998485_196.1 2021-02-11T09:58:17.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:17.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:17.029 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:17.071 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:17.071 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:17.306 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037401.998485_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:41.998485Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:58:17.306 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037470.242184_221.1 target=1_1613037401.998485_196.1 2021-02-11T09:58:22.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:22.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:22.308 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:22.349 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:22.349 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:22.593 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037401.998485_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:41.998485Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:58:22.594 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037470.242184_221.1 target=1_1613037401.998485_196.1 2021-02-11T09:58:27.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:27.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:27.595 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:27.635 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:27.635 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:27.876 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037401.998485_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:41.998485Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:58:27.877 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037470.242184_221.1 target=1_1613037401.998485_196.1 2021-02-11T09:58:32.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:32.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:32.877 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:32.918 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:32.918 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:33.160 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037401.998485_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:41.998485Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:58:33.161 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037470.242184_221.1 target=1_1613037401.998485_196.1 2021-02-11T09:58:38.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:38.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:38.162 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:38.203 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:38.204 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:38.457 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037401.998485_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:41.998485Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:58:38.457 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037470.242184_221.1 target=1_1613037401.998485_196.1 2021-02-11T09:58:43.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:43.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:43.459 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:43.499 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:43.499 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:43.757 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037401.998485_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:41.998485Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:58:43.757 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037470.242184_221.1 target=1_1613037401.998485_196.1 2021-02-11T09:58:48.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:48.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:48.759 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:48.802 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:48.802 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:49.045 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037401.998485_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:41.998485Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:58:49.046 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037470.242184_221.1 target=1_1613037401.998485_196.1 2021-02-11T09:58:50.431 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2021-02-11T09:58:50.427+0000 7f85567f4700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:58:50.432 INFO:tasks.rgw.c1.client.1.smithi026.stdout:2021-02-11T09:58:50.425+0000 7fd64cff9700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:58:54.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:54.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:54.047 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:54.087 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:54.088 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:54.330 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:58:54.331 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:58:54.331 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:58:54.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:54.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:54.331 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:54.371 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:54.371 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:54.634 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:58:54.635 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:58:54.635 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:58:54.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:54.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:54.635 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:54.673 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:54.673 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:54.928 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:58:54.929 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:58:54.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=a3c4910a-4529-4294-aed4-ef409f209cf5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:54.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=a3c4910a-4529-4294-aed4-ef409f209cf5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:54.945 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=a3c4910a-4529-4294-aed4-ef409f209cf5 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:58:55.006 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:55.007 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:55.039 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:58:55.035+0000 7f45c00f2080 0 failed reading obj info from .rgw.root:zone_info.30c2afa9-98cd-42de-acef-f4a612d65cea: (2) No such file or directory 2021-02-11T09:58:55.039 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:58:55.035+0000 7f45c00f2080 0 WARNING: could not read zone params for zone id=30c2afa9-98cd-42de-acef-f4a612d65cea name=test-zone3 2021-02-11T09:58:55.040 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:58:55.035+0000 7f45c00f2080 0 failed reading obj info from .rgw.root:zone_info.a3c4910a-4529-4294-aed4-ef409f209cf5: (2) No such file or directory 2021-02-11T09:58:55.040 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:58:55.035+0000 7f45c00f2080 0 WARNING: could not read zone params for zone id=a3c4910a-4529-4294-aed4-ef409f209cf5 name=test-zone1 2021-02-11T09:58:55.053 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"DHA6OWMQ42PGF1T7","secret_key":"xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T09:58:55.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:55.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:55.053 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:58:55.134 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:55.134 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:55.372 INFO:teuthology.orchestra.run.smithi133.stderr:Sending period to new master zone a3c4910a-4529-4294-aed4-ef409f209cf5 2021-02-11T09:58:55.480 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","epoch":3,"predecessor_uuid":"8ead83cf-fd6c-4023-bc3d-a277ca9366d6","sync_status":["","1_1613037321.168858_350.1","",""],"period_map":{"id":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","zones":[{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5"},{"id":"30c2afa9-98cd-42de-acef-f4a612d65cea","name":"test-zone3","endpoints":["http://smithi026.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"a3c4910a-4529-4294-aed4-ef409f209cf5","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","val":3299024572},{"key":"30c2afa9-98cd-42de-acef-f4a612d65cea","val":2754623789},{"key":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","val":3684997273},{"key":"a3c4910a-4529-4294-aed4-ef409f209cf5","val":2693415762}]},"master_zonegroup":"test-zonegroup","master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","realm_name":"test-realm","realm_epoch":4} 2021-02-11T09:58:55.480 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "a3c4910a-4529-4294-aed4-ef409f209cf5" 2021-02-11T09:59:55.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:59:55.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:59:55.831 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:59:55.873 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:59:55.873 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:59:55.903 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:59:55.902+0000 7f9809d84080 0 failed reading obj info from .rgw.root:zone_info.30c2afa9-98cd-42de-acef-f4a612d65cea: (2) No such file or directory 2021-02-11T09:59:55.903 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:59:55.902+0000 7f9809d84080 0 WARNING: could not read zone params for zone id=30c2afa9-98cd-42de-acef-f4a612d65cea name=test-zone3 2021-02-11T09:59:55.904 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:59:55.902+0000 7f9809d84080 0 failed reading obj info from .rgw.root:zone_info.a3c4910a-4529-4294-aed4-ef409f209cf5: (2) No such file or directory 2021-02-11T09:59:55.905 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T09:59:55.902+0000 7f9809d84080 0 WARNING: could not read zone params for zone id=a3c4910a-4529-4294-aed4-ef409f209cf5 name=test-zone1 2021-02-11T09:59:55.916 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"DHA6OWMQ42PGF1T7","secret_key":"xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T09:59:55.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:59:55.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:59:55.917 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:59:55.996 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:59:55.996 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:59:56.249 INFO:teuthology.orchestra.run.smithi133.stderr:Sending period to new master zone a3c4910a-4529-4294-aed4-ef409f209cf5 2021-02-11T09:59:56.358 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","epoch":4,"predecessor_uuid":"8ead83cf-fd6c-4023-bc3d-a277ca9366d6","sync_status":["","1_1613037321.168858_350.1","",""],"period_map":{"id":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","zones":[{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5"},{"id":"30c2afa9-98cd-42de-acef-f4a612d65cea","name":"test-zone3","endpoints":["http://smithi026.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"a3c4910a-4529-4294-aed4-ef409f209cf5","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","val":3299024572},{"key":"30c2afa9-98cd-42de-acef-f4a612d65cea","val":2754623789},{"key":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","val":3684997273},{"key":"a3c4910a-4529-4294-aed4-ef409f209cf5","val":2693415762}]},"master_zonegroup":"test-zonegroup","master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","realm_name":"test-realm","realm_epoch":4} 2021-02-11T09:59:56.359 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2021-02-11T10:00:56.360 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:00:56.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:00:56.361 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:00:56.404 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:00:56.404 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:00:56.432 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:00:56.430+0000 7f4f93472080 0 failed reading obj info from .rgw.root:zone_info.30c2afa9-98cd-42de-acef-f4a612d65cea: (2) No such file or directory 2021-02-11T10:00:56.432 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:00:56.430+0000 7f4f93472080 0 WARNING: could not read zone params for zone id=30c2afa9-98cd-42de-acef-f4a612d65cea name=test-zone3 2021-02-11T10:00:56.433 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:00:56.430+0000 7f4f93472080 0 failed reading obj info from .rgw.root:zone_info.a3c4910a-4529-4294-aed4-ef409f209cf5: (2) No such file or directory 2021-02-11T10:00:56.433 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:00:56.430+0000 7f4f93472080 0 WARNING: could not read zone params for zone id=a3c4910a-4529-4294-aed4-ef409f209cf5 name=test-zone1 2021-02-11T10:00:56.445 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"DHA6OWMQ42PGF1T7","secret_key":"xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T10:00:56.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:00:56.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:00:56.446 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:00:56.529 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:00:56.529 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:00:56.766 INFO:teuthology.orchestra.run.smithi133.stderr:Sending period to new master zone a3c4910a-4529-4294-aed4-ef409f209cf5 2021-02-11T10:00:56.855 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","epoch":5,"predecessor_uuid":"8ead83cf-fd6c-4023-bc3d-a277ca9366d6","sync_status":["","1_1613037321.168858_350.1","",""],"period_map":{"id":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","zones":[{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"30c2afa9-98cd-42de-acef-f4a612d65cea","name":"test-zone3","endpoints":["http://smithi026.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"a3c4910a-4529-4294-aed4-ef409f209cf5","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","val":3299024572},{"key":"30c2afa9-98cd-42de-acef-f4a612d65cea","val":2754623789},{"key":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","val":3684997273},{"key":"a3c4910a-4529-4294-aed4-ef409f209cf5","val":2693415762}]},"master_zonegroup":"test-zonegroup","master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","realm_name":"test-realm","realm_epoch":4} 2021-02-11T10:00:56.856 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2021-02-11T10:01:56.857 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ok 2021-02-11T10:01:56.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:56.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:56.859 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:56.899 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:56.899 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:57.123 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:01:57.153 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613037470.242184_221.1","last_update":"2021-02-11T09:57:50.242184Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:01:57.153 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:01:57.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:57.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:57.153 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:57.195 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:57.195 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:57.441 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:01:57.442 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:01:57.442 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:01:57.442 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:57.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:57.442 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:57.482 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:57.483 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:57.734 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:01:57.735 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:01:57.735 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:01:57.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:57.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:57.735 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:57.778 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:57.778 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:58.014 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:01:58.015 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:01:58.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'DHA6OWMQ42PGF1T7', '--secret', 'xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:01:58.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'DHA6OWMQ42PGF1T7', '--secret', 'xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:01:58.015 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key DHA6OWMQ42PGF1T7 --secret xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:01:58.107 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:58.108 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:58.171 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:01:58.169+0000 7f0e52844080 0 failed reading obj info from .rgw.root:zone_info.30c2afa9-98cd-42de-acef-f4a612d65cea: (2) No such file or directory 2021-02-11T10:01:58.171 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:01:58.169+0000 7f0e52844080 0 WARNING: could not read zone params for zone id=30c2afa9-98cd-42de-acef-f4a612d65cea name=test-zone3 2021-02-11T10:01:58.172 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:01:58.169+0000 7f0e52844080 0 failed reading obj info from .rgw.root:zone_info.a3c4910a-4529-4294-aed4-ef409f209cf5: (2) No such file or directory 2021-02-11T10:01:58.172 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:01:58.169+0000 7f0e52844080 0 WARNING: could not read zone params for zone id=a3c4910a-4529-4294-aed4-ef409f209cf5 name=test-zone1 2021-02-11T10:01:58.181 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"093010cd-c2ad-49e6-b90b-545e4affa824","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"DHA6OWMQ42PGF1T7","secret_key":"xJM1ca6wPpL4GTq4O7SFVlW5tViN4y6o"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","notif_pool":"remove.rgw.log:notif"} 2021-02-11T10:01:58.182 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:01:58.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:01:58.183 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:01:58.266 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:58.267 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:05.597 INFO:teuthology.orchestra.run.smithi133.stderr:Sending period to new master zone a3c4910a-4529-4294-aed4-ef409f209cf5 2021-02-11T10:02:05.705 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","epoch":6,"predecessor_uuid":"8ead83cf-fd6c-4023-bc3d-a277ca9366d6","sync_status":["","1_1613037321.168858_350.1","",""],"period_map":{"id":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","zones":[{"id":"093010cd-c2ad-49e6-b90b-545e4affa824","name":"remove","endpoints":[],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"30c2afa9-98cd-42de-acef-f4a612d65cea","name":"test-zone3","endpoints":["http://smithi026.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"a3c4910a-4529-4294-aed4-ef409f209cf5","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"093010cd-c2ad-49e6-b90b-545e4affa824","val":3212964688},{"key":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","val":3299024572},{"key":"30c2afa9-98cd-42de-acef-f4a612d65cea","val":2754623789},{"key":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","val":3684997273},{"key":"a3c4910a-4529-4294-aed4-ef409f209cf5","val":2693415762}]},"master_zonegroup":"test-zonegroup","master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","realm_name":"test-realm","realm_epoch":4} 2021-02-11T10:02:05.706 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:05.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:05.706 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:02:05.751 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:05.751 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:05.774 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:02:05.769+0000 7fd2973ed080 0 failed reading obj info from .rgw.root:zone_info.2f5f3613-ec60-4b48-a757-91ddd4a3ec48: (2) No such file or directory 2021-02-11T10:02:05.774 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:02:05.769+0000 7fd2973ed080 0 WARNING: could not read zone params for zone id=2f5f3613-ec60-4b48-a757-91ddd4a3ec48 name=test-zone2 2021-02-11T10:02:05.776 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:02:05.769+0000 7fd2973ed080 0 failed reading obj info from .rgw.root:zone_info.57f4d9d8-b2db-4f2f-b58c-d908c07c1a88: (2) No such file or directory 2021-02-11T10:02:05.776 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:02:05.769+0000 7fd2973ed080 0 WARNING: could not read zone params for zone id=57f4d9d8-b2db-4f2f-b58c-d908c07c1a88 name=test-zone4 2021-02-11T10:02:05.787 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","zones":[{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"30c2afa9-98cd-42de-acef-f4a612d65cea","name":"test-zone3","endpoints":["http://smithi026.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"a3c4910a-4529-4294-aed4-ef409f209cf5","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","sync_policy":{"groups":[]}} 2021-02-11T10:02:05.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:05.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:05.788 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:02:05.864 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:05.864 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:05.884 INFO:teuthology.orchestra.run.smithi026.stderr:zone name remove not found in zonegroup test-zonegroup 2021-02-11T10:02:05.890 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-02-11T10:02:05.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:05.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:05.891 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:02:05.926 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:05.926 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:05.967 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:02:05.964+0000 7f1ce43f0080 0 failed reading obj info from .rgw.root:zone_info.30c2afa9-98cd-42de-acef-f4a612d65cea: (2) No such file or directory 2021-02-11T10:02:05.967 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:02:05.964+0000 7f1ce43f0080 0 WARNING: could not read zone params for zone id=30c2afa9-98cd-42de-acef-f4a612d65cea name=test-zone3 2021-02-11T10:02:05.968 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:02:05.964+0000 7f1ce43f0080 0 failed reading obj info from .rgw.root:zone_info.a3c4910a-4529-4294-aed4-ef409f209cf5: (2) No such file or directory 2021-02-11T10:02:05.968 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:02:05.964+0000 7f1ce43f0080 0 WARNING: could not read zone params for zone id=a3c4910a-4529-4294-aed4-ef409f209cf5 name=test-zone1 2021-02-11T10:02:05.972 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:02:05.968+0000 7f1ce43f0080 0 zone id 093010cd-c2ad-49e6-b90b-545e4affa824 is not a part of zonegroup default 2021-02-11T10:02:05.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:05.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:05.982 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:02:06.028 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:06.028 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:06.084 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:02:06.077+0000 7f75396f8080 0 WARNING: can't generate connection for zone 093010cd-c2ad-49e6-b90b-545e4affa824 id remove: no endpoints defined 2021-02-11T10:02:06.416 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","epoch":7,"predecessor_uuid":"8ead83cf-fd6c-4023-bc3d-a277ca9366d6","sync_status":["","1_1613037321.168858_350.1","",""],"period_map":{"id":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","zones":[{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"30c2afa9-98cd-42de-acef-f4a612d65cea","name":"test-zone3","endpoints":["http://smithi026.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"a3c4910a-4529-4294-aed4-ef409f209cf5","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","val":3299024572},{"key":"30c2afa9-98cd-42de-acef-f4a612d65cea","val":2754623789},{"key":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","val":3684997273},{"key":"a3c4910a-4529-4294-aed4-ef409f209cf5","val":2693415762}]},"master_zonegroup":"test-zonegroup","master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","realm_name":"test-realm","realm_epoch":4} 2021-02-11T10:02:06.416 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2021-02-11T10:02:06.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:06.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:06.417 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:02:06.497 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:06.497 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:06.536 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"798b2d31-824b-4050-a1fd-008e99dbb0d8","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","sync_policy":{"groups":[]}} 2021-02-11T10:02:06.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:06.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:06.537 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:02:06.618 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:06.618 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:06.686 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"7d3d3377-7081-4a8c-aa77-72d5aff9368f","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","notif_pool":"remove.rgw.log:notif"} 2021-02-11T10:02:06.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:06.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:06.687 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:02:06.770 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:06.771 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:07.100 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","epoch":8,"predecessor_uuid":"8ead83cf-fd6c-4023-bc3d-a277ca9366d6","sync_status":["","1_1613037321.168858_350.1","",""],"period_map":{"id":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","zonegroups":[{"id":"798b2d31-824b-4050-a1fd-008e99dbb0d8","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"7d3d3377-7081-4a8c-aa77-72d5aff9368f","zones":[{"id":"7d3d3377-7081-4a8c-aa77-72d5aff9368f","name":"remove","endpoints":[],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","sync_policy":{"groups":[]}},{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","zones":[{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"30c2afa9-98cd-42de-acef-f4a612d65cea","name":"test-zone3","endpoints":["http://smithi026.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"a3c4910a-4529-4294-aed4-ef409f209cf5","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","val":3299024572},{"key":"30c2afa9-98cd-42de-acef-f4a612d65cea","val":2754623789},{"key":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","val":3684997273},{"key":"7d3d3377-7081-4a8c-aa77-72d5aff9368f","val":3804057480},{"key":"a3c4910a-4529-4294-aed4-ef409f209cf5","val":2693415762}]},"master_zonegroup":"test-zonegroup","master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","realm_name":"test-realm","realm_epoch":4} 2021-02-11T10:02:07.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:07.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:07.101 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:02:07.189 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:07.189 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:07.263 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:02:07.257+0000 7f13767e3080 0 zone id 7d3d3377-7081-4a8c-aa77-72d5aff9368f is not a part of zonegroup test-zonegroup 2021-02-11T10:02:07.264 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:02:07.257+0000 7f13767e3080 0 zone id 7d3d3377-7081-4a8c-aa77-72d5aff9368f is not a part of zonegroup default 2021-02-11T10:02:07.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:07.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:07.283 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:02:07.366 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:07.366 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:07.412 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:02:07.405+0000 7f064111e080 0 ERROR: zonegroup remove_zg has a non existent master zone 2021-02-11T10:02:07.418 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-02-11T10:02:07.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:07.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:07.418 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:02:07.499 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:07.499 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:07.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:07.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:07.537 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:02:07.620 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:07.620 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:07.942 INFO:teuthology.orchestra.run.smithi026.stdout:{"id":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","epoch":9,"predecessor_uuid":"8ead83cf-fd6c-4023-bc3d-a277ca9366d6","sync_status":["","1_1613037321.168858_350.1","",""],"period_map":{"id":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","zones":[{"id":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"30c2afa9-98cd-42de-acef-f4a612d65cea","name":"test-zone3","endpoints":["http://smithi026.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"a3c4910a-4529-4294-aed4-ef409f209cf5","name":"test-zone1","endpoints":["http://smithi026.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"2f5f3613-ec60-4b48-a757-91ddd4a3ec48","val":3299024572},{"key":"30c2afa9-98cd-42de-acef-f4a612d65cea","val":2754623789},{"key":"57f4d9d8-b2db-4f2f-b58c-d908c07c1a88","val":3684997273},{"key":"a3c4910a-4529-4294-aed4-ef409f209cf5","val":2693415762}]},"master_zonegroup":"test-zonegroup","master_zone":"a3c4910a-4529-4294-aed4-ef409f209cf5","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"52cf47b0-3540-4b08-a664-7ef37f763895","realm_name":"test-realm","realm_epoch":4} 2021-02-11T10:02:07.943 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2021-02-11T10:02:07.943 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-50 2021-02-11T10:02:08.456 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-51 2021-02-11T10:02:08.578 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-52 2021-02-11T10:02:08.671 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-53 2021-02-11T10:02:08.729 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-54 2021-02-11T10:02:09.847 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-55 2021-02-11T10:02:13.090 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2021-02-11T10:02:13.091 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-56 2021-02-11T10:02:13.163 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-57 2021-02-11T10:02:13.575 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-58 2021-02-11T10:02:13.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:13.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:13.649 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:13.683 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:13.684 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:13.911 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:02:13.949 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613037733.624243_334.1","last_update":"2021-02-11T10:02:13.624243Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:02:13.949 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:02:13.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:13.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:13.949 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:13.990 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:13.990 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:14.250 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:02:14.251 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037733.624243_334.1 target=1_1613037470.242184_221.1 2021-02-11T10:02:19.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:19.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:19.251 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:19.292 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:19.292 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:19.557 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:02:19.558 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037733.624243_334.1 target=1_1613037470.242184_221.1 2021-02-11T10:02:24.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:24.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:24.559 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:24.603 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:24.604 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:24.855 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:02:24.856 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037733.624243_334.1 target=1_1613037470.242184_221.1 2021-02-11T10:02:29.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:32.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:32.632 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:32.670 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:32.670 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:32.921 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:02:32.921 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037733.624243_334.1 target=1_1613037470.242184_221.1 2021-02-11T10:02:37.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:37.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:37.923 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:37.964 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:37.964 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:38.226 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:02:38.226 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037733.624243_334.1 target=1_1613037470.242184_221.1 2021-02-11T10:02:43.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:43.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:43.228 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:43.267 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:43.268 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:43.531 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:02:43.531 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037733.624243_334.1 target=1_1613037470.242184_221.1 2021-02-11T10:02:48.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:48.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:48.533 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:48.575 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:48.575 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:48.817 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:02:48.818 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037733.624243_334.1 target=1_1613037470.242184_221.1 2021-02-11T10:02:53.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:53.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:53.819 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:53.862 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:53.862 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:54.121 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:02:54.122 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037733.624243_334.1 target=1_1613037470.242184_221.1 2021-02-11T10:02:59.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:59.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:59.123 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:59.165 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:59.165 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:59.411 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:02:59.411 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037733.624243_334.1 target=1_1613037470.242184_221.1 2021-02-11T10:03:04.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:04.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:04.413 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:03:04.454 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:04.454 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:04.715 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:03:33.603 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037733.624243_334.1 target=1_1613037470.242184_221.1 2021-02-11T10:03:38.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:38.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:38.605 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:03:38.645 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:38.645 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:38.917 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:03:38.918 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037733.624243_334.1 target=1_1613037470.242184_221.1 2021-02-11T10:03:43.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:43.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:43.919 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:03:43.961 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:43.961 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:44.224 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:03:44.224 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037733.624243_334.1 target=1_1613037470.242184_221.1 2021-02-11T10:03:49.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:49.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:49.225 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:03:49.266 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:49.266 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:49.508 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:03:49.509 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037733.624243_334.1 target=1_1613037470.242184_221.1 2021-02-11T10:03:54.509 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:54.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:54.510 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:03:54.558 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:54.559 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:54.806 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:03:54.807 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037733.624243_334.1 target=1_1613037470.242184_221.1 2021-02-11T10:03:59.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:59.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:59.809 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:03:59.850 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:59.851 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:00.099 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:00.100 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037733.624243_334.1 target=1_1613037470.242184_221.1 2021-02-11T10:04:05.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:05.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:05.102 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:05.142 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:05.143 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:05.395 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:05.395 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037733.624243_334.1 target=1_1613037470.242184_221.1 2021-02-11T10:04:10.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:10.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:10.396 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:10.438 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:10.438 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:10.671 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037470.242184_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:57:50.242184Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:10.671 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037733.624243_334.1 target=1_1613037470.242184_221.1 2021-02-11T10:04:15.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:15.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:15.672 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:15.709 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:15.710 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:15.972 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037733.624243_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:13.624243Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:15.973 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:04:15.973 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:04:15.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:15.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:15.974 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:16.015 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:16.015 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:16.287 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037733.624243_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:13.624243Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:16.287 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:04:16.287 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:04:16.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:16.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:16.288 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:16.328 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:16.329 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:16.576 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037733.624243_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:13.624243Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:16.576 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:04:16.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'erivkc-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:04:16.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'erivkc-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:04:16.577 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket erivkc-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:04:16.620 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:16.620 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:16.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'erivkc-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:04:16.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'erivkc-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:04:16.966 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket erivkc-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:04:17.006 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:17.006 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:17.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'erivkc-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:04:17.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'erivkc-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:04:17.334 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket erivkc-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:04:17.416 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:17.416 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:17.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:17.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:17.747 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:17.832 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:17.832 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:18.101 INFO:teuthology.orchestra.run.smithi026.stdout:["obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:5","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:7","datalog.sync-status.shard.30c2afa9-98cd-42de-acef-f4a612d65cea.2","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-13:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.13:1","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23:8","obj_delete_at_hint.0000000026","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.5148.1-erivkc-44","obj_delete_at_hint.0000000028","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-38:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.38:10","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-46:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.3:8","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-48:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.5:8","obj_delete_at_hint.0000000040","meta.full-sync.index.0","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:3","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:1","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23:8","obj_delete_at_hint.0000000015","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.5148.4-erivkc-47","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22-erivkc-22","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20:8","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:3","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:1","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19:8","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:8","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:5","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19:5","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:3","obj_delete_at_hint.0000000024","2021-02-11-10-a3c4910a-4529-4294-aed4-ef409f209cf5.5479.4-erivkc-53","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.7-erivkc-7","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-15:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.15:1","data_log.3","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.27-erivkc-27","datalog.sync-status.shard.30c2afa9-98cd-42de-acef-f4a612d65cea.1","data_loggenerations_metadata","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.5350.1-erivkc-49","data.full-sync.index.2f5f3613-ec60-4b48-a757-91ddd4a3ec48.1","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:8","obj_delete_at_hint.0000000052","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-46:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.3:8","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-13:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.13:1","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24:8","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.38-erivkc-38","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:1","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","2021-02-11-10-a3c4910a-4529-4294-aed4-ef409f209cf5.5479.5-erivkc-54","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","2021-02-11-10-a3c4910a-4529-4294-aed4-ef409f209cf5.5479.8-erivkc-57","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bilog.trim","datalog.sync-status.shard.2f5f3613-ec60-4b48-a757-91ddd4a3ec48.3","obj_delete_at_hint.0000000016","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17-erivkc-17","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.40-erivkc-40","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.3-erivkc-3","obj_delete_at_hint.0000000041","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:5","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:1","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:5","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","obj_delete_at_hint.0000000014","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19:5","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-48:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.5:8","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:7","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:8","obj_delete_at_hint.0000000071","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:8","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22:8","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:3","obj_delete_at_hint.0000000074","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-36:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.36:10","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-49:a3c4910a-4529-4294-aed4-ef409f209cf5.5350.1:5","data.full-sync.index.2f5f3613-ec60-4b48-a757-91ddd4a3ec48.3","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","datalog.sync-status.shard.30c2afa9-98cd-42de-acef-f4a612d65cea.3","obj_delete_at_hint.0000000121","data_log.1","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-36:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.36:10","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.9-erivkc-9","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.8-erivkc-8","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21:8","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:7","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:3","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.41-erivkc-41","data.full-sync.index.30c2afa9-98cd-42de-acef-f4a612d65cea.0","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24-erivkc-24","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","data.full-sync.index.30c2afa9-98cd-42de-acef-f4a612d65cea.2","obj_delete_at_hint.0000000065","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.35-erivkc-35","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","datalog.sync-status.shard.2f5f3613-ec60-4b48-a757-91ddd4a3ec48.0","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.5148.3-erivkc-46","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.5148.5-erivkc-48","mdlog.sync-status.shard.2","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:1","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","obj_delete_at_hint.0000000116","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.39-erivkc-39","meta.log.f4d7bfad-1742-43a0-adcc-1b0cde4c6a67.1","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-37:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.37:10","mdlog.sync-status.shard.1","datalog.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:7","obj_delete_at_hint.0000000066","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-14:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.14:1","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10-erivkc-10","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:3","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.29-erivkc-29","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:3","datalog.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48","meta.full-sync.index.3","obj_delete_at_hint.0000000118","2021-02-11-09-2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.2-erivkc-43","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:1","obj_delete_at_hint.0000000036","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25:8","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25-erivkc-25","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-49:a3c4910a-4529-4294-aed4-ef409f209cf5.5350.1:5","obj_delete_at_hint.0000000119","datalog.sync-status.shard.2f5f3613-ec60-4b48-a757-91ddd4a3ec48.2","obj_delete_at_hint.0000000098","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.30-erivkc-30","mdlog.sync-status.shard.0","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:8","obj_delete_at_hint.0000000027","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26-erivkc-26","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.5-erivkc-5","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-15:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.15:1","mdlog.sync-status","obj_delete_at_hint.0000000019","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.5148.2-erivkc-45","obj_delete_at_hint.0000000039","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12-erivkc-12","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.36-erivkc-36","obj_delete_at_hint.0000000100","obj_delete_at_hint.0000000093","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:1","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11-erivkc-11","obj_delete_at_hint.0000000057","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26:8","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:1","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18-erivkc-18","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.32-erivkc-32","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:3","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23-erivkc-23","data_log.0","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.37-erivkc-37","obj_delete_at_hint.0000000012","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:3","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-47:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.4:8","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","2021-02-11-10-a3c4910a-4529-4294-aed4-ef409f209cf5.5479.2-erivkc-51","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.14-erivkc-14","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21:8","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.31-erivkc-31","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:3","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:8","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19:8","obj_delete_at_hint.0000000025","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.13-erivkc-13","obj_delete_at_hint.0000000123","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:3","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25:8","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:1","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","2021-02-11-10-a3c4910a-4529-4294-aed4-ef409f209cf5.5479.3-erivkc-52","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","data.full-sync.index.2f5f3613-ec60-4b48-a757-91ddd4a3ec48.2","obj_delete_at_hint.0000000013","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20:8","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:7","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.6-erivkc-6","obj_delete_at_hint.0000000051","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-47:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.4:8","obj_delete_at_hint.0000000106","data.full-sync.index.30c2afa9-98cd-42de-acef-f4a612d65cea.3","2021-02-11-09-2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.1-erivkc-42","2021-02-11-10-a3c4910a-4529-4294-aed4-ef409f209cf5.5350.1-erivkc-49","obj_delete_at_hint.0000000017","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.1-erivkc-1","data.full-sync.index.30c2afa9-98cd-42de-acef-f4a612d65cea.1","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-37:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.37:10","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:7","obj_delete_at_hint.0000000023","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21-erivkc-21","obj_delete_at_hint.0000000107","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.34-erivkc-34","2021-02-11-10-a3c4910a-4529-4294-aed4-ef409f209cf5.5479.6-erivkc-55","obj_delete_at_hint.0000000046","datalog.sync-status.shard.2f5f3613-ec60-4b48-a757-91ddd4a3ec48.1","meta.full-sync.index.2","2021-02-11-10-a3c4910a-4529-4294-aed4-ef409f209cf5.5479.7-erivkc-56","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:1","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:1","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26:8","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.2-erivkc-2","data_log.2","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-14:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.14:1","obj_delete_at_hint.0000000067","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.33-erivkc-33","obj_delete_at_hint.0000000086","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24:8","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.28-erivkc-28","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.15-erivkc-15","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22:8","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:1","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:3","2021-02-11-10-a3c4910a-4529-4294-aed4-ef409f209cf5.5479.1-erivkc-50","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:5","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16-erivkc-16","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19-erivkc-19","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","2021-02-11-10-a3c4910a-4529-4294-aed4-ef409f209cf5.5479.9-erivkc-58","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","datalog.sync-status.shard.30c2afa9-98cd-42de-acef-f4a612d65cea.0","obj_delete_at_hint.0000000042","data.full-sync.index.2f5f3613-ec60-4b48-a757-91ddd4a3ec48.0","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.4-erivkc-4","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:5","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20-erivkc-20","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-38:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.38:10"] 2021-02-11T10:04:18.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:18.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:18.133 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:18.173 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:18.173 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:18.439 INFO:teuthology.orchestra.run.smithi133.stdout:["obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:8","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","datalog.sync-status.shard.30c2afa9-98cd-42de-acef-f4a612d65cea.2","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-13:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.13:1","data.full-sync.index.a3c4910a-4529-4294-aed4-ef409f209cf5.0","obj_delete_at_hint.0000000026","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.5148.1-erivkc-44","obj_delete_at_hint.0000000028","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-38:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.38:10","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-46:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.3:8","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-48:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.5:8","obj_delete_at_hint.0000000040","meta.full-sync.index.0","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:3","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:1","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23:8","obj_delete_at_hint.0000000015","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.5148.4-erivkc-47","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22-erivkc-22","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:3","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19:8","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:5","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19:5","datalog.sync-status.shard.a3c4910a-4529-4294-aed4-ef409f209cf5.1","obj_delete_at_hint.0000000024","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.7-erivkc-7","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-15:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.15:1","data_log.3","datalog.sync-status.shard.30c2afa9-98cd-42de-acef-f4a612d65cea.1","data_loggenerations_metadata","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.5350.1-erivkc-49","datalog.sync-status.shard.a3c4910a-4529-4294-aed4-ef409f209cf5.2","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:8","obj_delete_at_hint.0000000052","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.38-erivkc-38","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-38:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.38:10","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20:8","2021-02-11-10-a3c4910a-4529-4294-aed4-ef409f209cf5.5479.5-erivkc-54","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","2021-02-11-10-a3c4910a-4529-4294-aed4-ef409f209cf5.5479.8-erivkc-57","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:8","bilog.trim","obj_delete_at_hint.0000000016","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17-erivkc-17","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.40-erivkc-40","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.3-erivkc-3","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22:8","data.full-sync.index.a3c4910a-4529-4294-aed4-ef409f209cf5.2","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:7","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:8","obj_delete_at_hint.0000000071","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:8","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:3","obj_delete_at_hint.0000000074","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-36:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.36:10","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:3","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","datalog.sync-status.shard.30c2afa9-98cd-42de-acef-f4a612d65cea.3","obj_delete_at_hint.0000000121","data_log.1","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.9-erivkc-9","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.8-erivkc-8","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:7","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-49:a3c4910a-4529-4294-aed4-ef409f209cf5.5350.1:5","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:1","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-47:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.4:8","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:3","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.41-erivkc-41","datalog.sync-status.shard.a3c4910a-4529-4294-aed4-ef409f209cf5.3","data.full-sync.index.30c2afa9-98cd-42de-acef-f4a612d65cea.0","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24-erivkc-24","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:1","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","data.full-sync.index.a3c4910a-4529-4294-aed4-ef409f209cf5.3","data.full-sync.index.30c2afa9-98cd-42de-acef-f4a612d65cea.2","obj_delete_at_hint.0000000065","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:8","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19:5","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.5148.3-erivkc-46","data.full-sync.index.a3c4910a-4529-4294-aed4-ef409f209cf5.1","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:1","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.5148.5-erivkc-48","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-37:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.37:10","mdlog.sync-status.shard.2","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:1","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","obj_delete_at_hint.0000000116","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24:8","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.39-erivkc-39","meta.log.f4d7bfad-1742-43a0-adcc-1b0cde4c6a67.1","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","datalog.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","obj_delete_at_hint.0000000066","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-14:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.14:1","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10-erivkc-10","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:3","meta.full-sync.index.3","obj_delete_at_hint.0000000118","2021-02-11-09-2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.2-erivkc-43","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:1","obj_delete_at_hint.0000000036","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-48:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.5:8","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25-erivkc-25","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-49:a3c4910a-4529-4294-aed4-ef409f209cf5.5350.1:5","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-46:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.3:8","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26-erivkc-26","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.5-erivkc-5","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-15:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.15:1","mdlog.sync-status","obj_delete_at_hint.0000000019","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.5148.2-erivkc-45","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:5","obj_delete_at_hint.0000000039","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23:8","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12-erivkc-12","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.36-erivkc-36","obj_delete_at_hint.0000000100","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:7","obj_delete_at_hint.0000000093","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:5","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:1","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11-erivkc-11","obj_delete_at_hint.0000000057","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:1","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18-erivkc-18","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:7","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23-erivkc-23","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21:8","data_log.0","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.37-erivkc-37","obj_delete_at_hint.0000000012","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","2021-02-11-10-a3c4910a-4529-4294-aed4-ef409f209cf5.5479.2-erivkc-51","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:1","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.14-erivkc-14","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21:8","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.31-erivkc-31","datalog.sync-status.shard.a3c4910a-4529-4294-aed4-ef409f209cf5.0","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-36:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.36:10","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","obj_delete_at_hint.0000000025","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.13-erivkc-13","obj_delete_at_hint.0000000123","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:3","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25:8","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:3","obj_delete_at_hint.0000000013","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20:8","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:3","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:1","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.6-erivkc-6","obj_delete_at_hint.0000000051","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-47:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.4:8","obj_delete_at_hint.0000000106","data.full-sync.index.30c2afa9-98cd-42de-acef-f4a612d65cea.3","2021-02-11-09-2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.1-erivkc-42","2021-02-11-10-a3c4910a-4529-4294-aed4-ef409f209cf5.5350.1-erivkc-49","obj_delete_at_hint.0000000017","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:1","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.1-erivkc-1","data.full-sync.index.30c2afa9-98cd-42de-acef-f4a612d65cea.1","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-37:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.37:10","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:7","obj_delete_at_hint.0000000023","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21-erivkc-21","obj_delete_at_hint.0000000107","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.34-erivkc-34","obj_delete_at_hint.0000000046","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-13:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.13:1","meta.full-sync.index.2","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:1","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:3","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26:8","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.2-erivkc-2","data_log.2","obj_delete_at_hint.0000000067","obj_delete_at_hint.0000000086","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24:8","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-14:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.14:1","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.28-erivkc-28","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.15-erivkc-15","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:3","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22:8","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25:8","obj_delete_at_hint.0000000050","datalog.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5","obj_delete_at_hint.0000000102","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:3","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:5","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16-erivkc-16","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19-erivkc-19","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19:8","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:7","obj_delete_at_hint.0000000097","datalog.sync-status.shard.30c2afa9-98cd-42de-acef-f4a612d65cea.0","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26:8","obj_delete_at_hint.0000000042","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.4-erivkc-4","bucket.sync-status.30c2afa9-98cd-42de-acef-f4a612d65cea:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:5","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:5","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20-erivkc-20"] 2021-02-11T10:04:18.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:18.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:18.481 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:18.515 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:18.515 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:18.715 INFO:teuthology.orchestra.run.smithi026.stdout:["obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:5","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:8","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:7","data.full-sync.index.a3c4910a-4529-4294-aed4-ef409f209cf5.0","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23:8","obj_delete_at_hint.0000000026","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.5148.1-erivkc-44","obj_delete_at_hint.0000000028","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.5148.4-erivkc-47","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22-erivkc-22","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20:8","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:1","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:8","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:3","datalog.sync-status.shard.a3c4910a-4529-4294-aed4-ef409f209cf5.1","obj_delete_at_hint.0000000024","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.7-erivkc-7","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-15:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.15:1","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-15:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.15:1","data_log.3","data_loggenerations_metadata","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.5350.1-erivkc-49","datalog.sync-status.shard.a3c4910a-4529-4294-aed4-ef409f209cf5.2","data.full-sync.index.2f5f3613-ec60-4b48-a757-91ddd4a3ec48.1","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-46:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.3:8","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-13:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.13:1","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24:8","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.38-erivkc-38","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-38:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.38:10","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:1","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20:8","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:8","bilog.trim","datalog.sync-status.shard.2f5f3613-ec60-4b48-a757-91ddd4a3ec48.3","obj_delete_at_hint.0000000016","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17-erivkc-17","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.40-erivkc-40","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.3-erivkc-3","meta.log.8ead83cf-fd6c-4023-bc3d-a277ca9366d6.1","obj_delete_at_hint.0000000041","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:5","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:1","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:5","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22:8","data.full-sync.index.a3c4910a-4529-4294-aed4-ef409f209cf5.2","obj_delete_at_hint.0000000014","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19:5","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-48:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.5:8","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22:8","obj_delete_at_hint.0000000074","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-49:a3c4910a-4529-4294-aed4-ef409f209cf5.5350.1:5","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:3","data.full-sync.index.2f5f3613-ec60-4b48-a757-91ddd4a3ec48.3","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-36:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.36:10","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.9-erivkc-9","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.8-erivkc-8","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21:8","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-49:a3c4910a-4529-4294-aed4-ef409f209cf5.5350.1:5","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:1","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-47:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.4:8","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.41-erivkc-41","datalog.sync-status.shard.a3c4910a-4529-4294-aed4-ef409f209cf5.3","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24-erivkc-24","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:1","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","data.full-sync.index.a3c4910a-4529-4294-aed4-ef409f209cf5.3","obj_delete_at_hint.0000000065","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:8","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.35-erivkc-35","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19:5","datalog.sync-status.shard.2f5f3613-ec60-4b48-a757-91ddd4a3ec48.0","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.5148.3-erivkc-46","data.full-sync.index.a3c4910a-4529-4294-aed4-ef409f209cf5.1","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:1","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.5148.5-erivkc-48","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-37:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.37:10","mdlog.sync-status.shard.2","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","obj_delete_at_hint.0000000116","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24:8","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.39-erivkc-39","meta.log.f4d7bfad-1742-43a0-adcc-1b0cde4c6a67.1","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-37:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.37:10","mdlog.sync-status.shard.1","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:7","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10-erivkc-10","meta.log.f2716509-104d-423b-ba5c-db1bd74642ab.1","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:3","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.29-erivkc-29","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","datalog.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48","meta.full-sync.index.3","obj_delete_at_hint.0000000118","2021-02-11-09-2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.2-erivkc-43","obj_delete_at_hint.0000000036","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25:8","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-48:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.5:8","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25-erivkc-25","obj_delete_at_hint.0000000119","datalog.sync-status.shard.2f5f3613-ec60-4b48-a757-91ddd4a3ec48.2","obj_delete_at_hint.0000000098","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-46:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.3:8","mdlog.sync-status.shard.0","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:8","obj_delete_at_hint.0000000027","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26-erivkc-26","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.5-erivkc-5","mdlog.sync-status","obj_delete_at_hint.0000000019","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.5148.2-erivkc-45","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:5","obj_delete_at_hint.0000000039","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23:8","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12-erivkc-12","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.36-erivkc-36","obj_delete_at_hint.0000000100","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:7","obj_delete_at_hint.0000000093","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:5","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11-erivkc-11","obj_delete_at_hint.0000000057","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26:8","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18-erivkc-18","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.32-erivkc-32","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:3","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:7","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23-erivkc-23","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21:8","data_log.0","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.37-erivkc-37","obj_delete_at_hint.0000000012","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:3","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-47:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.4:8","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:1","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.14-erivkc-14","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","datalog.sync-status.shard.a3c4910a-4529-4294-aed4-ef409f209cf5.0","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-36:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.36:10","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:3","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:8","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19:8","obj_delete_at_hint.0000000025","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.13-erivkc-13","obj_delete_at_hint.0000000123","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:1","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","2021-02-11-10-a3c4910a-4529-4294-aed4-ef409f209cf5.5479.3-erivkc-52","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:3","data.full-sync.index.2f5f3613-ec60-4b48-a757-91ddd4a3ec48.2","obj_delete_at_hint.0000000013","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:3","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:7","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:1","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.6-erivkc-6","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","2021-02-11-09-2f5f3613-ec60-4b48-a757-91ddd4a3ec48.5103.1-erivkc-42","2021-02-11-10-a3c4910a-4529-4294-aed4-ef409f209cf5.5350.1-erivkc-49","obj_delete_at_hint.0000000017","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:1","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.1-erivkc-1","obj_delete_at_hint.0000000023","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21-erivkc-21","meta.log.f2716509-104d-423b-ba5c-db1bd74642ab.0","obj_delete_at_hint.0000000107","2021-02-11-10-a3c4910a-4529-4294-aed4-ef409f209cf5.5479.6-erivkc-55","obj_delete_at_hint.0000000046","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-13:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.13:1","datalog.sync-status.shard.2f5f3613-ec60-4b48-a757-91ddd4a3ec48.1","meta.full-sync.index.2","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:3","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:1","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.2-erivkc-2","data_log.2","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-14:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.14:1","obj_delete_at_hint.0000000067","obj_delete_at_hint.0000000086","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-14:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.14:1","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.15-erivkc-15","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:3","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:1","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25:8","obj_delete_at_hint.0000000050","datalog.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5","obj_delete_at_hint.0000000102","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:3","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:3","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16-erivkc-16","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19-erivkc-19","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","2021-02-11-10-a3c4910a-4529-4294-aed4-ef409f209cf5.5479.9-erivkc-58","obj_delete_at_hint.0000000061","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19:8","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:7","obj_delete_at_hint.0000000097","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26:8","obj_delete_at_hint.0000000042","data.full-sync.index.2f5f3613-ec60-4b48-a757-91ddd4a3ec48.0","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.4-erivkc-4","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:5","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","2021-02-11-09-a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20-erivkc-20","bucket.sync-status.2f5f3613-ec60-4b48-a757-91ddd4a3ec48:erivkc-38:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.38:10"] 2021-02-11T10:04:18.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:18.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:18.752 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:18.789 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:18.789 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:19.033 INFO:teuthology.orchestra.run.smithi133.stdout:["obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:8","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","data.full-sync.index.a3c4910a-4529-4294-aed4-ef409f209cf5.0","obj_delete_at_hint.0000000026","obj_delete_at_hint.0000000028","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","datalog.sync-status.shard.a3c4910a-4529-4294-aed4-ef409f209cf5.1","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-15:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.15:1","data_log.3","data_loggenerations_metadata","datalog.sync-status.shard.a3c4910a-4529-4294-aed4-ef409f209cf5.2","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-38:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.38:10","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-20:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.20:8","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:8","bilog.trim","obj_delete_at_hint.0000000016","meta.log.8ead83cf-fd6c-4023-bc3d-a277ca9366d6.1","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-22:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.22:8","data.full-sync.index.a3c4910a-4529-4294-aed4-ef409f209cf5.2","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:3","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-49:a3c4910a-4529-4294-aed4-ef409f209cf5.5350.1:5","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:1","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-47:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.4:8","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","datalog.sync-status.shard.a3c4910a-4529-4294-aed4-ef409f209cf5.3","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:1","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","data.full-sync.index.a3c4910a-4529-4294-aed4-ef409f209cf5.3","obj_delete_at_hint.0000000065","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:8","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19:5","data.full-sync.index.a3c4910a-4529-4294-aed4-ef409f209cf5.1","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:1","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-37:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.37:10","mdlog.sync-status.shard.2","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","obj_delete_at_hint.0000000116","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-24:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.24:8","meta.log.f4d7bfad-1742-43a0-adcc-1b0cde4c6a67.1","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","meta.log.f2716509-104d-423b-ba5c-db1bd74642ab.1","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","meta.full-sync.index.3","obj_delete_at_hint.0000000118","obj_delete_at_hint.0000000036","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-48:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.5:8","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-46:a3c4910a-4529-4294-aed4-ef409f209cf5.5148.3:8","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:5","obj_delete_at_hint.0000000039","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-23:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.23:8","obj_delete_at_hint.0000000100","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:7","obj_delete_at_hint.0000000093","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:5","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:7","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-21:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.21:8","data_log.0","obj_delete_at_hint.0000000012","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:1","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","datalog.sync-status.shard.a3c4910a-4529-4294-aed4-ef409f209cf5.0","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-36:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.36:10","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-11:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.11:3","obj_delete_at_hint.0000000013","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:3","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:1","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:1","obj_delete_at_hint.0000000023","meta.log.f2716509-104d-423b-ba5c-db1bd74642ab.0","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-13:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.13:1","meta.full-sync.index.2","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-10:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.10:3","data_log.2","obj_delete_at_hint.0000000067","obj_delete_at_hint.0000000086","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-14:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.14:1","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-16:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.16:3","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-25:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.25:8","obj_delete_at_hint.0000000050","datalog.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5","obj_delete_at_hint.0000000102","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-18:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.18:3","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-19:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.19:8","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-12:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.12:7","obj_delete_at_hint.0000000097","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-26:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.26:8","obj_delete_at_hint.0000000042","bucket.sync-status.a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-17:a3c4910a-4529-4294-aed4-ef409f209cf5.4342.17:5","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030"] 2021-02-11T10:04:19.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:19.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:19.067 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:19.107 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:19.107 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:19.396 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037857.727581_378.3","last_update":"2021-02-11T10:04:17.727581Z"},{"marker":"1_1613037857.727581_445.3","last_update":"2021-02-11T10:04:17.727581Z"},{"marker":"1_1613037857.727581_205.3","last_update":"2021-02-11T10:04:17.727581Z"},{"marker":"1_1613037857.727581_344.2","last_update":"2021-02-11T10:04:17.727581Z"}] 2021-02-11T10:04:19.397 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:04:19.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:19.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:19.397 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:19.440 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:19.440 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:19.701 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037403.017566_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:43.017566Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037596.361413_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:59:56.361413Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037251.805432_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:54:11.805432Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037100.066735_115.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:51:40.066735Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:04:19.702 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1613037857.727581_378.3 target=1_1613037403.017566_293.1, shard=1 master=1_1613037857.727581_445.3 target=1_1613037596.361413_321.1, shard=2 master=1_1613037857.727581_205.3 target=1_1613037251.805432_108.1, shard=3 master=1_1613037857.727581_344.2 target=1_1613037100.066735_115.3 2021-02-11T10:04:24.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:24.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:24.703 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:24.745 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:24.745 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:25.015 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037403.017566_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:43.017566Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037596.361413_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:59:56.361413Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037251.805432_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:54:11.805432Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037100.066735_115.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:51:40.066735Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:04:25.015 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1613037857.727581_378.3 target=1_1613037403.017566_293.1, shard=1 master=1_1613037857.727581_445.3 target=1_1613037596.361413_321.1, shard=2 master=1_1613037857.727581_205.3 target=1_1613037251.805432_108.1, shard=3 master=1_1613037857.727581_344.2 target=1_1613037100.066735_115.3 2021-02-11T10:04:30.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:30.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:30.017 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:30.059 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:30.060 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:30.318 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037403.017566_293.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:43.017566Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037596.361413_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:59:56.361413Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037251.805432_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:54:11.805432Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037100.066735_115.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:51:40.066735Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:04:30.319 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1613037857.727581_378.3 target=1_1613037403.017566_293.1, shard=1 master=1_1613037857.727581_445.3 target=1_1613037596.361413_321.1, shard=2 master=1_1613037857.727581_205.3 target=1_1613037251.805432_108.1, shard=3 master=1_1613037857.727581_344.2 target=1_1613037100.066735_115.3 2021-02-11T10:04:35.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:35.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:35.320 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:35.361 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:35.361 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:35.622 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037857.727581_378.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.727581Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037857.727581_445.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.727581Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037857.727581_205.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.727581Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037857.727581_344.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.727581Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:04:35.622 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:04:35.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:35.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:35.623 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:35.661 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:35.662 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:35.930 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037857.727581_378.3","last_update":"2021-02-11T10:04:17.727581Z"},{"marker":"1_1613037857.727581_445.3","last_update":"2021-02-11T10:04:17.727581Z"},{"marker":"1_1613037857.727581_205.3","last_update":"2021-02-11T10:04:17.727581Z"},{"marker":"1_1613037857.727581_344.2","last_update":"2021-02-11T10:04:17.727581Z"}] 2021-02-11T10:04:35.931 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T10:04:35.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:35.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:35.931 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:36.013 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:36.013 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:36.271 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7045610624405686142},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037857.727581_378.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.727581Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037857.727581_445.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.727581Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037857.727581_205.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.727581Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037857.727581_344.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.727581Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:04:36.271 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T10:04:36.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:36.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:36.272 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:36.352 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:36.353 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:36.627 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037857.727581_378.3","last_update":"2021-02-11T10:04:17.727581Z"},{"marker":"1_1613037857.727581_445.3","last_update":"2021-02-11T10:04:17.727581Z"},{"marker":"1_1613037857.727581_205.3","last_update":"2021-02-11T10:04:17.727581Z"},{"marker":"1_1613037857.727581_344.2","last_update":"2021-02-11T10:04:17.727581Z"}] 2021-02-11T10:04:36.628 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T10:04:36.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:36.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:36.628 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:36.670 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:36.670 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:36.920 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1531294025241915737},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037857.727581_378.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.727581Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037857.727581_445.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.727581Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037857.727581_205.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.727581Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037857.727581_344.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.727581Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:04:36.920 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T10:04:36.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:36.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:36.921 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:37.001 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:37.001 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:37.266 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1613037858.302502_367.1","last_update":"2021-02-11T10:04:18.302502Z"},{"marker":"1_1613037858.305126_470.1","last_update":"2021-02-11T10:04:18.305126Z"},{"marker":"1_1613037858.307183_236.1","last_update":"2021-02-11T10:04:18.307183Z"},{"marker":"1_1613037858.300115_458.1","last_update":"2021-02-11T10:04:18.300115Z"}] 2021-02-11T10:04:37.267 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T10:04:37.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:37.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:37.267 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:37.310 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:37.310 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:37.575 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3492294689851219663},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037858.302502_367.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:18.302502Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037858.305126_470.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:18.305126Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037858.307183_236.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:18.307183Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037858.300115_458.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:18.300115Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:04:37.576 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T10:04:37.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:37.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:37.577 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:37.617 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:37.617 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:37.870 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1613037858.302502_367.1","last_update":"2021-02-11T10:04:18.302502Z"},{"marker":"1_1613037858.305126_470.1","last_update":"2021-02-11T10:04:18.305126Z"},{"marker":"1_1613037858.307183_236.1","last_update":"2021-02-11T10:04:18.307183Z"},{"marker":"1_1613037858.300115_458.1","last_update":"2021-02-11T10:04:18.300115Z"}] 2021-02-11T10:04:37.871 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T10:04:37.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:37.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:37.871 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:37.915 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:37.915 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:38.166 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7442380677805818242},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037858.302502_367.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:18.302502Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037858.305126_470.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:18.305126Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037858.307183_236.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:18.307183Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037858.300115_458.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:18.300115Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:04:38.166 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T10:04:38.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:38.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:38.167 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:38.246 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:38.247 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:38.507 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037857.914129_354.1","last_update":"2021-02-11T10:04:17.914129Z"},{"marker":"1_1613037857.917236_291.1","last_update":"2021-02-11T10:04:17.917236Z"},{"marker":"1_1613037857.921758_146.1","last_update":"2021-02-11T10:04:17.921758Z"},{"marker":"1_1613037857.907367_284.1","last_update":"2021-02-11T10:04:17.907367Z"}] 2021-02-11T10:04:38.508 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T10:04:38.508 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:38.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:38.508 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:38.594 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:38.594 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:38.843 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14314976040552073418},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037857.914129_354.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.914129Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037857.917236_291.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.917236Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037857.921758_146.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.921758Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037857.907367_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.907367Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:04:38.843 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T10:04:38.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:38.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:38.844 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:38.878 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:38.878 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:39.136 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037857.914129_354.1","last_update":"2021-02-11T10:04:17.914129Z"},{"marker":"1_1613037857.917236_291.1","last_update":"2021-02-11T10:04:17.917236Z"},{"marker":"1_1613037857.921758_146.1","last_update":"2021-02-11T10:04:17.921758Z"},{"marker":"1_1613037857.907367_284.1","last_update":"2021-02-11T10:04:17.907367Z"}] 2021-02-11T10:04:39.136 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T10:04:39.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:39.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:39.136 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:39.179 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:39.179 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:39.433 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12589397483957797345},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037857.914129_354.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.914129Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037857.917236_291.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.917236Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037857.921758_146.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.921758Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037857.907367_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:17.907367Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:04:39.433 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T10:04:39.434 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... ok 2021-02-11T10:04:39.434 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-59 2021-02-11T10:04:39.484 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-60 2021-02-11T10:04:39.559 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-61 2021-02-11T10:04:39.655 INFO:tasks.rgw.c1.client.1.smithi026.stdout:2021-02-11T10:04:39.651+0000 7fd64bff7700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:04:39.657 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2021-02-11T10:04:39.658+0000 7f6219feb700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:04:39.658 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2021-02-11T10:04:39.658+0000 7f85557f2700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:04:40.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:40.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:40.219 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:40.263 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:40.263 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:40.474 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:04:40.518 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613037879.600415_364.1","last_update":"2021-02-11T10:04:39.600415Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:04:40.518 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:04:40.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:40.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:40.518 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:40.555 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:40.555 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:40.802 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037879.592638_362.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:39.592638Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:40.803 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037879.600415_364.1 target=1_1613037879.592638_362.1 2021-02-11T10:04:45.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:45.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:45.804 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:45.845 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:45.845 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:46.107 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037879.592638_362.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:39.592638Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:46.107 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613037879.600415_364.1 target=1_1613037879.592638_362.1 2021-02-11T10:04:51.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:52.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:52.956 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:52.998 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:52.998 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:53.265 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037879.600415_364.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:39.600415Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:53.265 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:04:53.265 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:04:53.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:53.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:53.266 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:53.307 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:53.307 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:53.558 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037879.600415_364.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:39.600415Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:53.558 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:04:53.558 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:04:53.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:53.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:53.558 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:53.598 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:53.598 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:53.834 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037879.600415_364.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:39.600415Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:53.835 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:04:53.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:53.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:53.835 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:53.923 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:53.923 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:54.176 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:04:54.174+0000 7fdb66e5b080 1 bucket sync caught up with source: 2021-02-11T10:04:54.177 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.64.5, , 00000000001.35.5, , 00000000001.46.5, , , 00000000001.36.5, , , ] 2021-02-11T10:04:54.177 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.64.5, , 00000000001.35.5, , 00000000001.46.5, , , 00000000001.36.5, , , ] 2021-02-11T10:04:54.177 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:04:54.174+0000 7fdb66e5b080 0 bucket checkpoint complete 2021-02-11T10:04:54.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:54.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:54.212 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:54.259 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:54.259 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:54.509 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:04:54.507+0000 7f37805a7080 1 bucket sync caught up with source: 2021-02-11T10:04:54.509 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.64.5, , 00000000001.35.5, , 00000000001.46.5, , , 00000000001.36.5, , , ] 2021-02-11T10:04:54.509 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.64.5, , 00000000001.35.5, , 00000000001.46.5, , , 00000000001.36.5, , , ] 2021-02-11T10:04:54.509 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:04:54.507+0000 7f37805a7080 0 bucket checkpoint complete 2021-02-11T10:04:54.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:54.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:54.540 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:54.586 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:54.586 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:54.804 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:04:54.802+0000 7f842d23f080 1 bucket sync caught up with source: 2021-02-11T10:04:54.804 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.64.5, , 00000000001.35.5, , 00000000001.46.5, , , 00000000001.36.5, , , ] 2021-02-11T10:04:54.804 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.64.5, , 00000000001.35.5, , 00000000001.46.5, , , 00000000001.36.5, , , ] 2021-02-11T10:04:54.804 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:04:54.802+0000 7f842d23f080 0 bucket checkpoint complete 2021-02-11T10:04:54.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:54.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:54.840 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:54.884 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:54.884 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:55.128 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:04:55.127+0000 7fa883e2f080 1 bucket sync caught up with source: 2021-02-11T10:04:55.129 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.77.5, , 00000000001.39.5, , 00000000001.54.5, , , 00000000001.38.5, , , ] 2021-02-11T10:04:55.129 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.77.5, , 00000000001.39.5, , 00000000001.54.5, , , 00000000001.38.5, , , ] 2021-02-11T10:04:55.129 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:04:55.127+0000 7fa883e2f080 0 bucket checkpoint complete 2021-02-11T10:04:55.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:55.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:55.167 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:55.214 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:55.214 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:55.463 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:04:55.459+0000 7f1330034080 1 bucket sync caught up with source: 2021-02-11T10:04:55.464 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.77.5, , 00000000001.39.5, , 00000000001.54.5, , , 00000000001.38.5, , , ] 2021-02-11T10:04:55.464 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.77.5, , 00000000001.39.5, , 00000000001.54.5, , , 00000000001.38.5, , , ] 2021-02-11T10:04:55.464 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:04:55.459+0000 7f1330034080 0 bucket checkpoint complete 2021-02-11T10:04:55.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:55.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:55.499 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:55.546 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:55.546 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:55.798 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:04:55.795+0000 7fe16d07a080 1 bucket sync caught up with source: 2021-02-11T10:04:55.798 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.86.5, , 00000000001.33.5, , 00000000001.50.5, , , 00000000001.31.5, , , ] 2021-02-11T10:04:55.799 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.86.5, , 00000000001.33.5, , 00000000001.50.5, , , 00000000001.31.5, , , ] 2021-02-11T10:04:55.799 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:04:55.795+0000 7fe16d07a080 0 bucket checkpoint complete 2021-02-11T10:04:55.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:55.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:55.833 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:55.879 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:55.879 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:56.115 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:04:56.114+0000 7f6f72f98080 1 bucket sync caught up with source: 2021-02-11T10:04:56.115 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.86.5, , 00000000001.33.5, , 00000000001.50.5, , , 00000000001.31.5, , , ] 2021-02-11T10:04:56.115 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.86.5, , 00000000001.33.5, , 00000000001.50.5, , , 00000000001.31.5, , , ] 2021-02-11T10:04:56.115 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:04:56.114+0000 7f6f72f98080 0 bucket checkpoint complete 2021-02-11T10:04:56.150 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-59 zones={test-zone2, test-zone1} 2021-02-11T10:04:56.747 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:56.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:56.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:56.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:56.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:56.755 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:56.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:56.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:56.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:56.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:56.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:56.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:56.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:57.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:57.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:57.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:57.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:57.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:57.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:57.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:57.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:57.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:57.841 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-59 zones={test-zone2, test-zone1} 2021-02-11T10:04:57.841 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-59 zones={test-zone3, test-zone1} 2021-02-11T10:04:58.182 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:58.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:58.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:58.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:58.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:58.189 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:58.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:58.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:58.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:58.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:58.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:58.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:58.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:58.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:58.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:58.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:58.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:58.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:58.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:58.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:59.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:59.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:59.262 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-59 zones={test-zone3, test-zone1} 2021-02-11T10:04:59.262 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-59 zones={test-zone3, test-zone2} 2021-02-11T10:04:59.808 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:59.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:59.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:59.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:59.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:59.814 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:59.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:59.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:59.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:59.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:59.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:59.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:59.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:00.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:00.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:00.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:00.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:00.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:00.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:00.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:00.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:00.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:00.889 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-59 zones={test-zone3, test-zone2} 2021-02-11T10:05:00.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:00.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:00.890 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:00.934 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:00.934 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:01.186 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:01.186+0000 7f0b5a8fc080 1 bucket sync caught up with source: 2021-02-11T10:05:01.186 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.71.5, , 00000000001.38.5, , 00000000001.37.5, , , 00000000001.37.5, , , ] 2021-02-11T10:05:01.186 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.71.5, , 00000000001.38.5, , 00000000001.37.5, , , 00000000001.37.5, , , ] 2021-02-11T10:05:01.186 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:01.186+0000 7f0b5a8fc080 0 bucket checkpoint complete 2021-02-11T10:05:01.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:01.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:01.221 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:01.270 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:01.270 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:01.506 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:01.503+0000 7f24c8303080 1 bucket sync caught up with source: 2021-02-11T10:05:01.506 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.71.5, , 00000000001.38.5, , 00000000001.37.5, , , 00000000001.37.5, , , ] 2021-02-11T10:05:01.506 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.71.5, , 00000000001.38.5, , 00000000001.37.5, , , 00000000001.37.5, , , ] 2021-02-11T10:05:01.506 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:01.503+0000 7f24c8303080 0 bucket checkpoint complete 2021-02-11T10:05:01.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:01.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:01.543 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:01.598 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:01.598 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:01.812 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:01.810+0000 7f9ce8b21080 1 bucket sync caught up with source: 2021-02-11T10:05:01.812 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.71.5, , 00000000001.38.5, , 00000000001.37.5, , , 00000000001.37.5, , , ] 2021-02-11T10:05:01.812 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.71.5, , 00000000001.38.5, , 00000000001.37.5, , , 00000000001.37.5, , , ] 2021-02-11T10:05:01.812 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:01.810+0000 7f9ce8b21080 0 bucket checkpoint complete 2021-02-11T10:05:01.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:01.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:01.847 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:01.893 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:01.893 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:02.142 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:02.139+0000 7f0aa697f080 1 bucket sync caught up with source: 2021-02-11T10:05:02.142 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.89.5, , 00000000001.36.5, , 00000000001.37.5, , , 00000000001.48.5, , , ] 2021-02-11T10:05:02.142 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.89.5, , 00000000001.36.5, , 00000000001.37.5, , , 00000000001.48.5, , , ] 2021-02-11T10:05:02.142 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:02.139+0000 7f0aa697f080 0 bucket checkpoint complete 2021-02-11T10:05:02.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:02.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:02.179 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:02.223 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:02.223 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:02.470 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:02.467+0000 7f8137d84080 1 bucket sync caught up with source: 2021-02-11T10:05:02.470 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.89.5, , 00000000001.36.5, , 00000000001.37.5, , , 00000000001.48.5, , , ] 2021-02-11T10:05:02.470 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.89.5, , 00000000001.36.5, , 00000000001.37.5, , , 00000000001.48.5, , , ] 2021-02-11T10:05:02.470 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:02.467+0000 7f8137d84080 0 bucket checkpoint complete 2021-02-11T10:05:02.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:02.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:02.507 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:02.603 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:02.604 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:02.847 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:02.843+0000 7f0786343080 1 bucket sync caught up with source: 2021-02-11T10:05:02.847 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.92.5, , 00000000001.33.5, , 00000000001.35.5, , , 00000000001.45.5, , , ] 2021-02-11T10:05:02.848 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.92.5, , 00000000001.33.5, , 00000000001.35.5, , , 00000000001.45.5, , , ] 2021-02-11T10:05:02.848 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:02.843+0000 7f0786343080 0 bucket checkpoint complete 2021-02-11T10:05:02.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:02.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:02.879 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:02.926 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:02.927 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:03.174 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:03.174+0000 7f88e8bde080 1 bucket sync caught up with source: 2021-02-11T10:05:03.174 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.92.5, , 00000000001.33.5, , 00000000001.35.5, , , 00000000001.45.5, , , ] 2021-02-11T10:05:03.175 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.92.5, , 00000000001.33.5, , 00000000001.35.5, , , 00000000001.45.5, , , ] 2021-02-11T10:05:03.175 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:03.174+0000 7f88e8bde080 0 bucket checkpoint complete 2021-02-11T10:05:03.211 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-60 zones={test-zone2, test-zone1} 2021-02-11T10:05:04.086 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:05:04.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:04.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:04.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:04.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:04.093 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:05:04.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:04.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:04.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:04.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:04.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:04.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:04.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:04.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:04.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:04.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:04.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:04.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:04.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:04.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:04.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:05.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:05.190 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-60 zones={test-zone2, test-zone1} 2021-02-11T10:05:05.190 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-60 zones={test-zone3, test-zone1} 2021-02-11T10:05:05.975 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:05:05.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:05.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:05.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:05.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:05.981 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:05:05.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:05.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:05.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:05.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:05.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:06.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:06.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:06.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:06.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:06.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:06.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:06.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:06.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:06.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:06.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:06.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:07.062 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-60 zones={test-zone3, test-zone1} 2021-02-11T10:05:07.062 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-60 zones={test-zone3, test-zone2} 2021-02-11T10:05:07.684 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:05:07.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:07.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:07.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:07.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:07.692 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:05:07.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:07.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:07.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:07.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:07.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:07.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:07.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:07.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:08.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:08.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:08.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:08.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:08.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:08.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:08.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:08.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:08.770 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-60 zones={test-zone3, test-zone2} 2021-02-11T10:05:08.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:08.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:08.770 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:08.817 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:08.817 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:09.069 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:09.066+0000 7f54042d3080 1 bucket sync caught up with source: 2021-02-11T10:05:09.069 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.74.5, , 00000000001.40.5, , 00000000001.38.5, , , 00000000001.34.5, , , ] 2021-02-11T10:05:09.069 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.74.5, , 00000000001.40.5, , 00000000001.38.5, , , 00000000001.34.5, , , ] 2021-02-11T10:05:09.069 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:09.066+0000 7f54042d3080 0 bucket checkpoint complete 2021-02-11T10:05:09.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:09.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:09.105 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:09.153 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:09.153 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:09.397 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:09.395+0000 7f37faa4a080 1 bucket sync caught up with source: 2021-02-11T10:05:09.397 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.74.5, , 00000000001.40.5, , 00000000001.38.5, , , 00000000001.34.5, , , ] 2021-02-11T10:05:09.397 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.74.5, , 00000000001.40.5, , 00000000001.38.5, , , 00000000001.34.5, , , ] 2021-02-11T10:05:09.397 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:09.395+0000 7f37faa4a080 0 bucket checkpoint complete 2021-02-11T10:05:09.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:09.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:09.427 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:09.475 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:09.475 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:09.681 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:09.678+0000 7f46c5b3c080 1 bucket sync caught up with source: 2021-02-11T10:05:09.681 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.74.5, , 00000000001.40.5, , 00000000001.38.5, , , 00000000001.34.5, , , ] 2021-02-11T10:05:09.681 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.74.5, , 00000000001.40.5, , 00000000001.38.5, , , 00000000001.34.5, , , ] 2021-02-11T10:05:09.681 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:09.678+0000 7f46c5b3c080 0 bucket checkpoint complete 2021-02-11T10:05:09.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:09.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:09.712 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:09.766 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:09.766 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:09.999 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:09.995+0000 7f8ff7c55080 1 bucket sync caught up with source: 2021-02-11T10:05:10.000 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.62.5, , 00000000001.58.5, , 00000000001.46.5, , , 00000000001.34.5, , , ] 2021-02-11T10:05:10.000 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.62.5, , 00000000001.58.5, , 00000000001.46.5, , , 00000000001.34.5, , , ] 2021-02-11T10:05:10.000 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:09.995+0000 7f8ff7c55080 0 bucket checkpoint complete 2021-02-11T10:05:10.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:10.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:10.034 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:10.082 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:10.082 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:10.321 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:10.319+0000 7f33c15a9080 1 bucket sync caught up with source: 2021-02-11T10:05:10.322 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.62.5, , 00000000001.58.5, , 00000000001.46.5, , , 00000000001.34.5, , , ] 2021-02-11T10:05:10.322 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.62.5, , 00000000001.58.5, , 00000000001.46.5, , , 00000000001.34.5, , , ] 2021-02-11T10:05:10.322 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:10.319+0000 7f33c15a9080 0 bucket checkpoint complete 2021-02-11T10:05:10.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:10.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:10.363 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:10.453 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:10.453 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:10.681 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:10.679+0000 7f231b88b080 1 bucket sync caught up with source: 2021-02-11T10:05:10.681 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.60.5, , 00000000001.54.5, , 00000000001.41.5, , , 00000000001.38.5, , , ] 2021-02-11T10:05:10.681 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.60.5, , 00000000001.54.5, , 00000000001.41.5, , , 00000000001.38.5, , , ] 2021-02-11T10:05:10.681 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:10.679+0000 7f231b88b080 0 bucket checkpoint complete 2021-02-11T10:05:10.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:10.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:10.711 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:10.760 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:10.760 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:11.008 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:11.006+0000 7f57e1289080 1 bucket sync caught up with source: 2021-02-11T10:05:11.008 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.60.5, , 00000000001.54.5, , 00000000001.41.5, , , 00000000001.38.5, , , ] 2021-02-11T10:05:11.008 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.60.5, , 00000000001.54.5, , 00000000001.41.5, , , 00000000001.38.5, , , ] 2021-02-11T10:05:11.008 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:11.006+0000 7f57e1289080 0 bucket checkpoint complete 2021-02-11T10:05:11.043 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-61 zones={test-zone2, test-zone1} 2021-02-11T10:05:12.346 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:05:12.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:12.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:12.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:12.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:12.354 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:05:12.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:12.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:12.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:12.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:12.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:12.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:12.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:12.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:12.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:12.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:12.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:12.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:13.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:13.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:13.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:13.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:13.495 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-61 zones={test-zone2, test-zone1} 2021-02-11T10:05:13.495 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-61 zones={test-zone3, test-zone1} 2021-02-11T10:05:13.625 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:05:13.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:13.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:13.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:13.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:13.631 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:05:13.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:13.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:13.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:13.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:13.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:13.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:13.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:13.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:14.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:14.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:14.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:14.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:14.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:14.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:14.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:14.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:14.834 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-61 zones={test-zone3, test-zone1} 2021-02-11T10:05:14.834 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-61 zones={test-zone3, test-zone2} 2021-02-11T10:05:15.181 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:05:15.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:15.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:15.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:15.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:15.189 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:05:15.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:15.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:15.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:15.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:15.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:15.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:15.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:15.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:15.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:15.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:15.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:15.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:15.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:16.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:16.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:16.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:16.274 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-61 zones={test-zone3, test-zone2} 2021-02-11T10:05:16.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'erivkc-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:05:16.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'erivkc-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:05:16.274 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket erivkc-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:05:16.315 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:16.315 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:16.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'erivkc-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:05:16.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'erivkc-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:05:16.654 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket erivkc-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:05:16.736 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:16.736 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:17.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'erivkc-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:05:17.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'erivkc-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:05:17.051 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket erivkc-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:05:17.134 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:17.134 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:17.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'erivkc-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:05:17.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'erivkc-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:05:17.473 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket erivkc-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:05:17.556 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:17.556 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:17.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'erivkc-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:05:17.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'erivkc-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:05:17.906 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket erivkc-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:05:17.983 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:17.984 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:18.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'erivkc-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:05:18.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'erivkc-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:05:18.295 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket erivkc-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:05:18.379 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:18.379 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:20.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:20.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:20.667 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:20.722 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:20.722 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:20.940 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:20.937+0000 7fb1d8c2f080 1 waiting for incremental sync to catch up: 2021-02-11T10:05:20.940 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000006.74.5, , 00000000006.47.5, 00000000004.44.1, 00000000006.56.5, , 00000000004.49.1, 00000000006.48.5, 00000000004.42.1, , 00000000004.37.1] 2021-02-11T10:05:20.940 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000006.74.5, 00000000004.75.1, 00000000006.47.5, 00000000004.44.1, 00000000006.56.5, 00000000004.54.1, 00000000004.49.1, 00000000006.48.5, 00000000004.42.1, 00000000004.66.1, 00000000004.37.1] 2021-02-11T10:05:25.944 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:25.941+0000 7fb1d8c2f080 1 waiting for incremental sync to catch up: 2021-02-11T10:05:30.019 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000006.74.5, , 00000000006.47.5, 00000000004.44.1, 00000000006.56.5, , 00000000004.49.1, 00000000006.48.5, 00000000004.42.1, , 00000000004.37.1] 2021-02-11T10:05:30.019 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000006.74.5, 00000000004.75.1, 00000000006.47.5, 00000000004.44.1, 00000000006.56.5, 00000000004.54.1, 00000000004.49.1, 00000000006.48.5, 00000000004.42.1, 00000000004.66.1, 00000000004.37.1] 2021-02-11T10:05:30.947 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:30.945+0000 7fb1d8c2f080 1 waiting for incremental sync to catch up: 2021-02-11T10:05:30.948 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000006.74.5, 00000000004.75.1, 00000000006.47.5, 00000000004.44.1, 00000000006.56.5, , 00000000004.49.1, 00000000006.48.5, 00000000004.42.1, , 00000000004.37.1] 2021-02-11T10:05:30.948 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000006.74.5, 00000000004.75.1, 00000000006.47.5, 00000000004.44.1, 00000000006.56.5, 00000000004.54.1, 00000000004.49.1, 00000000006.48.5, 00000000004.42.1, 00000000004.66.1, 00000000004.37.1] 2021-02-11T10:05:35.951 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:35.949+0000 7fb1d8c2f080 1 bucket sync caught up with source: 2021-02-11T10:05:35.952 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000006.74.5, 00000000004.75.1, 00000000006.47.5, 00000000004.44.1, 00000000006.56.5, 00000000004.54.1, 00000000004.49.1, 00000000006.48.5, 00000000004.42.1, 00000000004.66.1, 00000000004.37.1] 2021-02-11T10:05:35.952 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000006.74.5, 00000000004.75.1, 00000000006.47.5, 00000000004.44.1, 00000000006.56.5, 00000000004.54.1, 00000000004.49.1, 00000000006.48.5, 00000000004.42.1, 00000000004.66.1, 00000000004.37.1] 2021-02-11T10:05:35.952 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:35.949+0000 7fb1d8c2f080 0 bucket checkpoint complete 2021-02-11T10:05:35.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:35.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:35.988 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:36.036 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:36.036 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:36.285 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:36.282+0000 7faf087dd080 1 bucket sync caught up with source: 2021-02-11T10:05:36.285 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000006.74.5, 00000000004.75.1, 00000000006.47.5, 00000000004.44.1, 00000000006.56.5, 00000000004.54.1, 00000000004.49.1, 00000000006.48.5, 00000000004.42.1, 00000000004.66.1, 00000000004.37.1] 2021-02-11T10:05:36.286 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000006.74.5, 00000000004.75.1, 00000000006.47.5, 00000000004.44.1, 00000000006.56.5, 00000000004.54.1, 00000000004.49.1, 00000000006.48.5, 00000000004.42.1, 00000000004.66.1, 00000000004.37.1] 2021-02-11T10:05:36.286 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:36.282+0000 7faf087dd080 0 bucket checkpoint complete 2021-02-11T10:05:36.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:36.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:36.316 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:36.363 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:36.364 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:36.597 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:36.597+0000 7fa56b1e8080 1 bucket sync caught up with source: 2021-02-11T10:05:36.597 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000006.74.5, 00000000004.75.1, 00000000006.47.5, 00000000004.44.1, 00000000006.56.5, 00000000004.54.1, 00000000004.49.1, 00000000006.48.5, 00000000004.42.1, 00000000004.66.1, 00000000004.37.1] 2021-02-11T10:05:36.597 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000006.74.5, 00000000004.75.1, 00000000006.47.5, 00000000004.44.1, 00000000006.56.5, 00000000004.54.1, 00000000004.49.1, 00000000006.48.5, 00000000004.42.1, 00000000004.66.1, 00000000004.37.1] 2021-02-11T10:05:36.597 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:36.597+0000 7fa56b1e8080 0 bucket checkpoint complete 2021-02-11T10:05:36.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:36.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:36.631 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:36.681 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:36.681 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:36.926 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:36.922+0000 7f503bd20080 1 bucket sync caught up with source: 2021-02-11T10:05:36.926 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000002.83.5, , 00000000002.45.5, , 00000000002.60.5, , , 00000000002.44.5, , , ] 2021-02-11T10:05:36.926 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000002.83.5, , 00000000002.45.5, , 00000000002.60.5, , , 00000000002.44.5, , , ] 2021-02-11T10:05:36.926 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:36.922+0000 7f503bd20080 0 bucket checkpoint complete 2021-02-11T10:05:36.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:36.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:36.960 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:37.003 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:37.003 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:37.248 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:37.246+0000 7f4b9ca12080 1 bucket sync caught up with source: 2021-02-11T10:05:37.248 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000002.83.5, , 00000000002.45.5, , 00000000002.60.5, , , 00000000002.44.5, , , ] 2021-02-11T10:05:37.248 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000002.83.5, , 00000000002.45.5, , 00000000002.60.5, , , 00000000002.44.5, , , ] 2021-02-11T10:05:37.248 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:37.246+0000 7f4b9ca12080 0 bucket checkpoint complete 2021-02-11T10:05:37.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:37.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:37.285 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:37.376 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:37.376 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:37.622 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:37.618+0000 7f5f7cdde080 1 bucket sync caught up with source: 2021-02-11T10:05:37.622 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000002.92.5, , 00000000002.43.5, , 00000000002.56.5, , , 00000000002.41.5, , , ] 2021-02-11T10:05:37.622 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000002.92.5, , 00000000002.43.5, , 00000000002.56.5, , , 00000000002.41.5, , , ] 2021-02-11T10:05:37.622 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:37.618+0000 7f5f7cdde080 0 bucket checkpoint complete 2021-02-11T10:05:37.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:37.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:37.654 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:37.697 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:37.697 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:37.949 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:37.949+0000 7f7c8aafe080 1 bucket sync caught up with source: 2021-02-11T10:05:37.949 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000002.92.5, , 00000000002.43.5, , 00000000002.56.5, , , 00000000002.41.5, , , ] 2021-02-11T10:05:37.950 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000002.92.5, , 00000000002.43.5, , 00000000002.56.5, , , 00000000002.41.5, , , ] 2021-02-11T10:05:37.950 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:37.949+0000 7f7c8aafe080 0 bucket checkpoint complete 2021-02-11T10:05:37.995 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-59 zones={test-zone2, test-zone1} 2021-02-11T10:05:38.906 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:05:38.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:38.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:38.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:38.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:38.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:05:38.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:05:38.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:05:38.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:05:38.914 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:05:38.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:38.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:38.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:38.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:38.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:05:38.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:05:38.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:05:38.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:05:38.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:38.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:39.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:39.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:39.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:39.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:39.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:39.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:39.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:39.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:39.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:39.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:39.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:05:40.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:05:40.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:05:40.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:05:40.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:05:40.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:05:40.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:05:40.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:05:40.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:05:40.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:05:40.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:05:40.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:05:41.070 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-59 zones={test-zone2, test-zone1} 2021-02-11T10:05:41.070 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-59 zones={test-zone3, test-zone1} 2021-02-11T10:05:41.309 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:05:41.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:41.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:41.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:41.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:41.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:05:41.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:05:41.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:05:41.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:05:41.315 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:05:41.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:41.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:41.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:41.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:41.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:05:41.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:05:41.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:05:41.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:05:41.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:41.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:41.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:41.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:41.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:41.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:41.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:41.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:42.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:42.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:42.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:42.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:42.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:05:42.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:05:42.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:05:42.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:05:42.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:05:42.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:05:42.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:05:42.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:05:43.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:05:43.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:05:43.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:05:43.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:05:43.486 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-59 zones={test-zone3, test-zone1} 2021-02-11T10:05:43.486 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-59 zones={test-zone3, test-zone2} 2021-02-11T10:05:44.052 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:05:44.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:44.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:44.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:44.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:44.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:05:44.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:05:44.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:05:44.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:05:44.061 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:05:44.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:44.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:44.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:44.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:44.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:05:44.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:05:44.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:05:44.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:05:44.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:44.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:44.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:44.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:44.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:44.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:44.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:44.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:44.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:44.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:44.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:45.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:45.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:05:45.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:05:45.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:05:45.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:05:45.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:05:45.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:05:45.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:05:45.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:05:45.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:05:45.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:05:45.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:05:46.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:05:46.206 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-59 zones={test-zone3, test-zone2} 2021-02-11T10:05:46.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:46.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:46.206 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:46.252 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:46.252 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:46.482 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:46.481+0000 7f33ee6e8080 1 bucket sync caught up with source: 2021-02-11T10:05:46.483 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000006.81.5, 00000000004.70.1, 00000000006.50.5, 00000000004.38.1, 00000000007.50.3, 00000000004.71.1, 00000000004.44.1, 00000000006.43.5, 00000000004.72.1, 00000000004.50.1, 00000000004.79.1] 2021-02-11T10:05:46.483 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000006.81.5, 00000000004.70.1, 00000000006.50.5, 00000000004.38.1, 00000000007.50.3, 00000000004.71.1, 00000000004.44.1, 00000000006.43.5, 00000000004.72.1, 00000000004.50.1, 00000000004.79.1] 2021-02-11T10:05:46.483 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:46.481+0000 7f33ee6e8080 0 bucket checkpoint complete 2021-02-11T10:05:46.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:46.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:46.522 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:46.570 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:46.571 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:46.797 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:46.794+0000 7f10aa220080 1 bucket sync caught up with source: 2021-02-11T10:05:46.797 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000006.81.5, 00000000004.70.1, 00000000006.50.5, 00000000004.38.1, 00000000007.50.3, 00000000004.71.1, 00000000004.44.1, 00000000006.43.5, 00000000004.72.1, 00000000004.50.1, 00000000004.79.1] 2021-02-11T10:05:46.797 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000006.81.5, 00000000004.70.1, 00000000006.50.5, 00000000004.38.1, 00000000007.50.3, 00000000004.71.1, 00000000004.44.1, 00000000006.43.5, 00000000004.72.1, 00000000004.50.1, 00000000004.79.1] 2021-02-11T10:05:46.797 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:46.794+0000 7f10aa220080 0 bucket checkpoint complete 2021-02-11T10:05:46.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:46.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:46.832 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:46.881 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:46.881 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:47.106 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:47.105+0000 7f06fdc7c080 1 bucket sync caught up with source: 2021-02-11T10:05:47.107 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000006.81.5, 00000000004.70.1, 00000000006.50.5, 00000000004.38.1, 00000000007.50.3, 00000000004.71.1, 00000000004.44.1, 00000000006.43.5, 00000000004.72.1, 00000000004.50.1, 00000000004.79.1] 2021-02-11T10:05:47.107 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000006.81.5, 00000000004.70.1, 00000000006.50.5, 00000000004.38.1, 00000000007.50.3, 00000000004.71.1, 00000000004.44.1, 00000000006.43.5, 00000000004.72.1, 00000000004.50.1, 00000000004.79.1] 2021-02-11T10:05:47.107 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:47.105+0000 7f06fdc7c080 0 bucket checkpoint complete 2021-02-11T10:05:47.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:47.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:47.143 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:47.192 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:47.192 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:47.455 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:47.450+0000 7f998ba64080 1 bucket sync caught up with source: 2021-02-11T10:05:47.456 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000002.95.5, , 00000000002.46.5, , 00000000002.47.5, , , 00000000002.52.5, , , ] 2021-02-11T10:05:47.456 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000002.95.5, , 00000000002.46.5, , 00000000002.47.5, , , 00000000002.52.5, , , ] 2021-02-11T10:05:47.456 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:47.450+0000 7f998ba64080 0 bucket checkpoint complete 2021-02-11T10:05:47.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:47.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:47.496 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:47.545 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:47.545 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:47.807 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:47.802+0000 7f48814b1080 1 bucket sync caught up with source: 2021-02-11T10:05:47.807 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000002.95.5, , 00000000002.46.5, , 00000000002.47.5, , , 00000000002.52.5, , , ] 2021-02-11T10:05:47.807 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000002.95.5, , 00000000002.46.5, , 00000000002.47.5, , , 00000000002.52.5, , , ] 2021-02-11T10:05:47.807 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:47.802+0000 7f48814b1080 0 bucket checkpoint complete 2021-02-11T10:05:47.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:47.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:47.834 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:47.879 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:47.879 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:48.108 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:48.106+0000 7f4b193dc080 1 bucket sync caught up with source: 2021-02-11T10:05:48.108 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000004.100.5, 00000000002.66.1, 00000000004.43.5, 00000000002.36.1, 00000000004.45.5, 00000000002.53.1, 00000000002.57.1, 00000000004.51.5, 00000000002.56.1, 00000000002.50.1, 00000000002.89.1] 2021-02-11T10:05:48.108 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000004.100.5, 00000000002.66.1, 00000000004.43.5, 00000000002.36.1, 00000000004.45.5, 00000000002.53.1, 00000000002.57.1, 00000000004.51.5, 00000000002.56.1, 00000000002.50.1, 00000000002.89.1] 2021-02-11T10:05:48.108 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:48.106+0000 7f4b193dc080 0 bucket checkpoint complete 2021-02-11T10:05:48.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:48.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:48.141 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:48.188 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:48.188 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:48.421 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:48.417+0000 7f44311bb080 1 bucket sync caught up with source: 2021-02-11T10:05:48.421 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000004.100.5, 00000000002.66.1, 00000000004.43.5, 00000000002.36.1, 00000000004.45.5, 00000000002.53.1, 00000000002.57.1, 00000000004.51.5, 00000000002.56.1, 00000000002.50.1, 00000000002.89.1] 2021-02-11T10:05:48.421 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000004.100.5, 00000000002.66.1, 00000000004.43.5, 00000000002.36.1, 00000000004.45.5, 00000000002.53.1, 00000000002.57.1, 00000000004.51.5, 00000000002.56.1, 00000000002.50.1, 00000000002.89.1] 2021-02-11T10:05:48.421 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:48.417+0000 7f44311bb080 0 bucket checkpoint complete 2021-02-11T10:05:48.456 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-60 zones={test-zone2, test-zone1} 2021-02-11T10:05:49.695 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:05:49.704 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:49.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:49.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:49.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:49.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:05:49.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:05:49.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:05:49.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:05:49.705 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:05:49.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:49.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:49.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:49.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:49.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:05:49.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:05:49.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:05:49.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:05:49.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:49.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:49.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:49.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:50.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:50.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:50.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:50.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:50.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:50.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:50.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:50.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:50.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:05:50.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:05:50.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:05:51.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:05:51.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:05:51.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:05:51.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:05:51.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:05:51.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:05:51.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:05:51.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:05:51.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:05:51.849 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-60 zones={test-zone2, test-zone1} 2021-02-11T10:05:51.849 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-60 zones={test-zone3, test-zone1} 2021-02-11T10:05:52.046 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:05:52.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:52.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:52.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:52.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:52.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:05:52.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:05:52.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:05:52.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:05:52.052 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:05:52.057 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:52.057 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:52.057 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:52.057 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:52.057 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:05:52.057 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:05:52.057 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:05:52.057 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:05:52.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:52.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:52.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:52.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:52.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:52.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:52.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:52.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:52.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:52.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:52.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:53.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:53.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:05:53.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:05:53.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:05:53.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:05:53.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:05:53.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:05:53.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:05:53.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:05:53.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:05:53.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:05:54.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:05:54.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:05:54.265 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-60 zones={test-zone3, test-zone1} 2021-02-11T10:05:54.265 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-60 zones={test-zone3, test-zone2} 2021-02-11T10:05:55.179 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:05:55.186 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:55.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:55.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:55.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:55.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:05:55.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:05:55.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:05:55.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:05:55.187 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:05:55.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:05:55.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:05:55.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:05:55.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:05:55.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:05:55.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:05:55.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:05:55.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:05:55.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:55.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:55.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:05:55.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:55.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:55.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:05:55.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:55.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:55.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:05:56.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:56.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:56.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:05:56.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:05:56.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:05:56.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:05:56.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:05:56.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:05:56.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:05:56.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:05:56.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:05:56.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:05:57.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:05:57.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:05:57.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:05:57.374 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-60 zones={test-zone3, test-zone2} 2021-02-11T10:05:57.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:57.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:57.374 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:57.418 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:57.418 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:57.668 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:57.665+0000 7f77e3341080 1 bucket sync caught up with source: 2021-02-11T10:05:57.668 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000007.81.3, 00000000004.72.1, 00000000006.46.5, 00000000004.35.1, 00000000006.52.5, 00000000004.42.1, 00000000004.83.1, 00000000006.40.5, 00000000004.38.1, 00000000004.50.1, 00000000004.43.1] 2021-02-11T10:05:57.668 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000007.81.3, 00000000004.72.1, 00000000006.46.5, 00000000004.35.1, 00000000006.52.5, 00000000004.42.1, 00000000004.83.1, 00000000006.40.5, 00000000004.38.1, 00000000004.50.1, 00000000004.43.1] 2021-02-11T10:05:57.668 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:57.665+0000 7f77e3341080 0 bucket checkpoint complete 2021-02-11T10:05:57.704 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:57.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:57.704 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:57.751 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:57.751 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:57.970 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:57.966+0000 7fb3face8080 1 bucket sync caught up with source: 2021-02-11T10:05:57.970 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000007.81.3, 00000000004.72.1, 00000000006.46.5, 00000000004.35.1, 00000000006.52.5, 00000000004.42.1, 00000000004.83.1, 00000000006.40.5, 00000000004.38.1, 00000000004.50.1, 00000000004.43.1] 2021-02-11T10:05:57.970 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000007.81.3, 00000000004.72.1, 00000000006.46.5, 00000000004.35.1, 00000000006.52.5, 00000000004.42.1, 00000000004.83.1, 00000000006.40.5, 00000000004.38.1, 00000000004.50.1, 00000000004.43.1] 2021-02-11T10:05:57.970 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:57.966+0000 7fb3face8080 0 bucket checkpoint complete 2021-02-11T10:05:57.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:57.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:57.999 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:58.045 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:58.046 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:58.271 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:58.269+0000 7f39eb677080 1 bucket sync caught up with source: 2021-02-11T10:05:58.271 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000007.81.3, 00000000004.72.1, 00000000006.46.5, 00000000004.35.1, 00000000006.52.5, 00000000004.42.1, 00000000004.83.1, 00000000006.40.5, 00000000004.38.1, 00000000004.50.1, 00000000004.43.1] 2021-02-11T10:05:58.272 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000007.81.3, 00000000004.72.1, 00000000006.46.5, 00000000004.35.1, 00000000006.52.5, 00000000004.42.1, 00000000004.83.1, 00000000006.40.5, 00000000004.38.1, 00000000004.50.1, 00000000004.43.1] 2021-02-11T10:05:58.272 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:05:58.269+0000 7f39eb677080 0 bucket checkpoint complete 2021-02-11T10:05:58.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:58.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:05:58.307 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:05:58.355 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:58.355 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:58.593 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:58.590+0000 7fc40582a080 1 bucket sync caught up with source: 2021-02-11T10:05:58.593 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000002.66.5, , 00000000002.62.5, , 00000000002.54.5, , , 00000000002.38.5, , , ] 2021-02-11T10:05:58.594 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000002.66.5, , 00000000002.62.5, , 00000000002.54.5, , , 00000000002.38.5, , , ] 2021-02-11T10:05:58.594 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:05:58.590+0000 7fc40582a080 0 bucket checkpoint complete 2021-02-11T10:05:58.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:11.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:11.860 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:11.907 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:11.907 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:12.142 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:12.138+0000 7f5840d69080 1 bucket sync caught up with source: 2021-02-11T10:06:12.142 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000002.66.5, , 00000000002.62.5, , 00000000002.54.5, , , 00000000002.38.5, , , ] 2021-02-11T10:06:12.143 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000002.66.5, , 00000000002.62.5, , 00000000002.54.5, , , 00000000002.38.5, , , ] 2021-02-11T10:06:12.143 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:12.138+0000 7f5840d69080 0 bucket checkpoint complete 2021-02-11T10:06:12.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:12.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:12.179 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:12.266 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:12.266 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:12.502 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:12.498+0000 7f291686f080 1 bucket sync caught up with source: 2021-02-11T10:06:12.502 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000002.64.5, , 00000000002.58.5, , 00000000002.49.5, , , 00000000002.42.5, , , ] 2021-02-11T10:06:12.502 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000002.64.5, , 00000000002.58.5, , 00000000002.49.5, , , 00000000002.42.5, , , ] 2021-02-11T10:06:12.502 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:12.498+0000 7f291686f080 0 bucket checkpoint complete 2021-02-11T10:06:12.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:12.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:12.534 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:12.580 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:12.580 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:12.826 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:12.825+0000 7fe42fd4a080 1 bucket sync caught up with source: 2021-02-11T10:06:12.827 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000002.64.5, , 00000000002.58.5, , 00000000002.49.5, , , 00000000002.42.5, , , ] 2021-02-11T10:06:12.827 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000002.64.5, , 00000000002.58.5, , 00000000002.49.5, , , 00000000002.42.5, , , ] 2021-02-11T10:06:12.827 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:12.825+0000 7fe42fd4a080 0 bucket checkpoint complete 2021-02-11T10:06:12.862 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-61 zones={test-zone2, test-zone1} 2021-02-11T10:06:14.186 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:06:14.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:14.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:14.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:14.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:14.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:06:14.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:06:14.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:06:14.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:06:14.194 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:06:14.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:14.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:14.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:14.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:14.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:06:14.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:06:14.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:06:14.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:06:14.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:14.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:14.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:14.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:14.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:14.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:14.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:14.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:14.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:15.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:15.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:15.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:15.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:06:15.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:06:15.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:06:15.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:06:15.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:06:15.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:06:15.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:06:15.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:06:15.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:06:16.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:06:16.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:06:16.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:06:16.355 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-61 zones={test-zone2, test-zone1} 2021-02-11T10:06:16.355 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-61 zones={test-zone3, test-zone1} 2021-02-11T10:06:16.943 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:06:16.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:16.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:16.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:16.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:16.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:06:16.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:06:16.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:06:16.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:06:16.949 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:06:16.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:16.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:16.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:16.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:16.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:06:16.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:06:16.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:06:16.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:06:16.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:17.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:17.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:17.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:17.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:17.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:17.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:17.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:17.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:17.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:17.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:17.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:18.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:06:18.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:06:18.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:06:18.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:06:18.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:06:18.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:06:18.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:06:18.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:06:18.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:06:18.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:06:18.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:06:19.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:06:19.182 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-61 zones={test-zone3, test-zone1} 2021-02-11T10:06:19.182 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-61 zones={test-zone3, test-zone2} 2021-02-11T10:06:20.051 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:06:20.058 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:20.059 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:20.059 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:20.059 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:20.059 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:06:20.059 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:06:20.059 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:06:20.059 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:06:20.059 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:06:20.065 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:20.065 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:20.065 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:20.065 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:20.065 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:06:20.065 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:06:20.065 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:06:20.066 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:06:20.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:20.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:20.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:20.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:20.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:20.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:20.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:20.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:20.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:20.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:20.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:21.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:21.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:06:21.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:06:21.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:06:21.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:06:21.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:06:21.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:06:21.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:06:21.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:06:21.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:06:21.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:06:22.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:06:22.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:06:22.245 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-61 zones={test-zone3, test-zone2} 2021-02-11T10:06:22.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:22.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:22.246 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:22.287 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:22.287 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:22.558 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037920.844457_425.1","last_update":"2021-02-11T10:05:20.844457Z"},{"marker":"1_1613037918.690431_513.3","last_update":"2021-02-11T10:05:18.690431Z"},{"marker":"1_1613037920.842206_256.1","last_update":"2021-02-11T10:05:20.842206Z"},{"marker":"1_1613037920.799078_408.1","last_update":"2021-02-11T10:05:20.799078Z"}] 2021-02-11T10:06:22.558 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:06:22.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:22.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:22.559 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:22.598 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:22.598 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:22.847 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037920.844457_425.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:20.844457Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037918.690431_513.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:18.690431Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037920.842206_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:20.842206Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037920.799078_408.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:20.799078Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:06:22.847 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:06:22.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:22.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:22.848 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:22.886 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:22.886 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:23.146 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037920.844457_425.1","last_update":"2021-02-11T10:05:20.844457Z"},{"marker":"1_1613037918.690431_513.3","last_update":"2021-02-11T10:05:18.690431Z"},{"marker":"1_1613037920.842206_256.1","last_update":"2021-02-11T10:05:20.842206Z"},{"marker":"1_1613037920.799078_408.1","last_update":"2021-02-11T10:05:20.799078Z"}] 2021-02-11T10:06:23.146 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T10:06:23.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:23.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:23.146 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:23.232 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:23.232 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:23.502 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7045610624405686142},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037920.844457_425.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:20.844457Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037918.690431_513.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:18.690431Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037920.842206_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:20.842206Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037920.799078_408.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:20.799078Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:06:23.502 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T10:06:23.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:23.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:23.503 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:23.539 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:23.539 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:23.788 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037920.844457_425.1","last_update":"2021-02-11T10:05:20.844457Z"},{"marker":"1_1613037918.690431_513.3","last_update":"2021-02-11T10:05:18.690431Z"},{"marker":"1_1613037920.842206_256.1","last_update":"2021-02-11T10:05:20.842206Z"},{"marker":"1_1613037920.799078_408.1","last_update":"2021-02-11T10:05:20.799078Z"}] 2021-02-11T10:06:23.788 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T10:06:23.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:23.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:23.788 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:23.832 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:23.832 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:24.086 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1531294025241915737},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037920.844457_425.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:20.844457Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037918.690431_513.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:18.690431Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037920.842206_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:20.842206Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037920.799078_408.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:20.799078Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:06:24.086 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T10:06:24.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:24.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:24.087 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:24.168 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:24.168 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:24.439 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1613037920.804606_387.1","last_update":"2021-02-11T10:05:20.804606Z"},{"marker":"1_1613037858.305126_470.1","last_update":"2021-02-11T10:04:18.305126Z"},{"marker":"1_1613037920.799646_263.1","last_update":"2021-02-11T10:05:20.799646Z"},{"marker":"1_1613037920.847267_513.1","last_update":"2021-02-11T10:05:20.847267Z"}] 2021-02-11T10:06:24.439 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T10:06:24.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:24.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:24.440 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:24.488 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:24.488 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:24.750 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3492294689851219663},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037920.804606_387.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:20.804606Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037858.305126_470.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:18.305126Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037920.799646_263.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:20.799646Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037920.847267_513.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:20.847267Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:06:24.751 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T10:06:24.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:24.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:24.751 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:24.793 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:24.793 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:25.070 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1613037920.804606_387.1","last_update":"2021-02-11T10:05:20.804606Z"},{"marker":"1_1613037858.305126_470.1","last_update":"2021-02-11T10:04:18.305126Z"},{"marker":"1_1613037920.799646_263.1","last_update":"2021-02-11T10:05:20.799646Z"},{"marker":"1_1613037920.847267_513.1","last_update":"2021-02-11T10:05:20.847267Z"}] 2021-02-11T10:06:25.071 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T10:06:25.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:25.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:25.072 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:25.116 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:25.116 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:25.370 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7442380677805818242},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037920.804606_387.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:20.804606Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037858.305126_470.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:18.305126Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037920.799646_263.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:20.799646Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037920.847267_513.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:20.847267Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:06:25.371 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T10:06:25.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:25.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:25.371 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:25.408 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:25.409 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:25.665 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037926.278152_386.3","last_update":"2021-02-11T10:05:26.278152Z"},{"marker":"1_1613037926.278152_319.3","last_update":"2021-02-11T10:05:26.278152Z"},{"marker":"1_1613037926.278152_173.3","last_update":"2021-02-11T10:05:26.278152Z"},{"marker":"1_1613037926.278152_327.2","last_update":"2021-02-11T10:05:26.278152Z"}] 2021-02-11T10:06:25.665 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T10:06:25.665 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:25.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:25.665 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:25.752 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:25.753 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:26.013 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14314976040552073418},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037926.278152_386.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:26.278152Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037926.278152_319.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:26.278152Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037926.278152_173.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:26.278152Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037926.278152_327.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:26.278152Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:06:26.013 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T10:06:26.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:26.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:26.014 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:26.049 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:26.049 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:26.319 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613037926.278152_386.3","last_update":"2021-02-11T10:05:26.278152Z"},{"marker":"1_1613037926.278152_319.3","last_update":"2021-02-11T10:05:26.278152Z"},{"marker":"1_1613037926.278152_173.3","last_update":"2021-02-11T10:05:26.278152Z"},{"marker":"1_1613037926.278152_327.2","last_update":"2021-02-11T10:05:26.278152Z"}] 2021-02-11T10:06:26.320 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T10:06:26.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:26.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:26.320 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:26.361 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:26.361 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:26.621 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12589397483957797345},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037926.278152_386.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:26.278152Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037926.278152_319.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:26.278152Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037926.278152_173.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:26.278152Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037926.278152_327.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:26.278152Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:06:26.621 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T10:06:26.622 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2021-02-11T10:06:26.622 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-62 2021-02-11T10:06:26.825 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-63 2021-02-11T10:06:26.985 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-64 2021-02-11T10:06:28.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:28.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:28.327 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:28.369 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:28.369 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:28.562 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:06:28.608 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613037987.700483_416.1","last_update":"2021-02-11T10:06:27.700483Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:06:28.608 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:06:28.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:28.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:28.609 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:28.650 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:28.650 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:28.878 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037987.700483_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:06:27.700483Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:06:28.878 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:06:28.878 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:06:28.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:28.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:28.879 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:28.918 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:28.918 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:29.175 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037987.700483_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:06:27.700483Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:06:29.175 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:06:29.175 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:06:29.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:29.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:29.176 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:29.215 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:29.215 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:29.472 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613037987.700483_416.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:06:27.700483Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:06:29.472 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:06:29.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:29.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:29.473 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:29.561 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:29.561 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:29.780 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:29.776+0000 7f4da03c6080 1 bucket sync caught up with source: 2021-02-11T10:06:29.780 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.54.5, , 00000000001.77.5, , 00000000001.55.5, , , 00000000001.41.5, , , ] 2021-02-11T10:06:29.780 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.54.5, , 00000000001.77.5, , 00000000001.55.5, , , 00000000001.41.5, , , ] 2021-02-11T10:06:29.780 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:29.776+0000 7f4da03c6080 0 bucket checkpoint complete 2021-02-11T10:06:29.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:29.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:29.812 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:29.861 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:29.861 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:30.073+0000 7f545837a080 1 bucket sync caught up with source: 2021-02-11T10:06:30.079 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.54.5, , 00000000001.77.5, , 00000000001.55.5, , , 00000000001.41.5, , , ] 2021-02-11T10:06:30.079 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.54.5, , 00000000001.77.5, , 00000000001.55.5, , , 00000000001.41.5, , , ] 2021-02-11T10:06:30.079 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:30.073+0000 7f545837a080 0 bucket checkpoint complete 2021-02-11T10:06:30.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:30.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:30.110 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:30.156 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:30.157 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:30.385 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:30.380+0000 7fb3a0dd6080 1 bucket sync caught up with source: 2021-02-11T10:06:30.385 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.54.5, , 00000000001.77.5, , 00000000001.55.5, , , 00000000001.41.5, , , ] 2021-02-11T10:06:30.385 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.54.5, , 00000000001.77.5, , 00000000001.55.5, , , 00000000001.41.5, , , ] 2021-02-11T10:06:30.385 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:30.380+0000 7fb3a0dd6080 0 bucket checkpoint complete 2021-02-11T10:06:30.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:30.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:30.421 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:30.470 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:30.470 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:30.728 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:30.725+0000 7f7dae064080 1 bucket sync caught up with source: 2021-02-11T10:06:30.729 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.51.5, , 00000000001.86.5, , 00000000001.52.5, , , 00000000001.36.5, , , ] 2021-02-11T10:06:30.729 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.52.5, , , 00000000001.36.5, , , ] 2021-02-11T10:06:30.729 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:30.725+0000 7f7dae064080 0 bucket checkpoint complete 2021-02-11T10:06:30.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:30.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:30.765 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:30.809 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:30.809 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:31.018 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:31.013+0000 7eff84f36080 1 bucket sync caught up with source: 2021-02-11T10:06:31.018 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.51.5, , 00000000001.86.5, , 00000000001.52.5, , , 00000000001.36.5, , , ] 2021-02-11T10:06:31.019 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.52.5, , , 00000000001.36.5, , , ] 2021-02-11T10:06:31.019 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:31.013+0000 7eff84f36080 0 bucket checkpoint complete 2021-02-11T10:06:31.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:31.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:31.058 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:31.107 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:31.107 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:31.340 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:31.337+0000 7f809203c080 1 bucket sync caught up with source: 2021-02-11T10:06:31.341 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.51.5, , 00000000001.95.5, , 00000000001.49.5, , , 00000000001.41.5, , , ] 2021-02-11T10:06:31.341 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.51.5, , 00000000001.95.5, , 00000000001.49.5, , , 00000000001.41.5, , , ] 2021-02-11T10:06:31.341 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:31.337+0000 7f809203c080 0 bucket checkpoint complete 2021-02-11T10:06:31.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:31.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:31.373 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:31.420 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:31.420 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:31.662 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:31.660+0000 7f28b218b080 1 bucket sync caught up with source: 2021-02-11T10:06:31.662 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.51.5, , 00000000001.95.5, , 00000000001.49.5, , , 00000000001.41.5, , , ] 2021-02-11T10:06:31.662 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.51.5, , 00000000001.95.5, , 00000000001.49.5, , , 00000000001.41.5, , , ] 2021-02-11T10:06:31.663 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:31.660+0000 7f28b218b080 0 bucket checkpoint complete 2021-02-11T10:06:31.703 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-62 zones={test-zone2, test-zone1} 2021-02-11T10:06:32.458 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:06:32.467 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:32.467 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:32.467 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:32.467 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:32.467 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:06:32.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:32.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:32.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:32.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:32.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:32.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:32.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:32.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:32.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:32.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:33.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:33.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:33.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:33.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:33.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:33.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:33.566 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-62 zones={test-zone2, test-zone1} 2021-02-11T10:06:33.566 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-62 zones={test-zone3, test-zone1} 2021-02-11T10:06:33.589 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:06:33.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:33.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:33.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:33.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:33.594 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:06:33.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:33.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:33.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:33.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:33.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:33.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:33.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:33.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:33.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:34.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:34.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:34.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:34.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:34.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:34.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:34.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:34.669 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-62 zones={test-zone3, test-zone1} 2021-02-11T10:06:34.669 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-62 zones={test-zone3, test-zone2} 2021-02-11T10:06:35.659 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:06:35.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:35.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:35.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:35.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:35.667 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:06:35.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:35.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:35.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:35.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:35.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:35.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:35.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:35.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:35.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:36.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:36.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:36.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:36.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:36.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:36.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:36.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:36.758 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-62 zones={test-zone3, test-zone2} 2021-02-11T10:06:36.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:36.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:36.758 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:36.801 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:36.801 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:37.043 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:37.040+0000 7f2b05281080 1 bucket sync caught up with source: 2021-02-11T10:06:37.043 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.47.5, , 00000000001.23.5, , 00000000001.43.5, , , 00000000001.55.5, , , ] 2021-02-11T10:06:37.043 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.47.5, , 00000000001.23.5, , 00000000001.43.5, , , 00000000001.55.5, , , ] 2021-02-11T10:06:37.043 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:37.040+0000 7f2b05281080 0 bucket checkpoint complete 2021-02-11T10:06:37.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:37.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:37.082 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:37.130 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:37.130 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:37.371 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:37.365+0000 7ff2a03a6080 1 bucket sync caught up with source: 2021-02-11T10:06:37.372 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.47.5, , 00000000001.23.5, , 00000000001.43.5, , , 00000000001.55.5, , , ] 2021-02-11T10:06:37.372 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.47.5, , 00000000001.23.5, , 00000000001.43.5, , , 00000000001.55.5, , , ] 2021-02-11T10:06:37.372 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:37.365+0000 7ff2a03a6080 0 bucket checkpoint complete 2021-02-11T10:06:37.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:37.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:37.402 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:37.449 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:37.449 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:37.673 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:37.668+0000 7f33166ac080 1 bucket sync caught up with source: 2021-02-11T10:06:37.673 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.47.5, , 00000000001.23.5, , 00000000001.43.5, , , 00000000001.55.5, , , ] 2021-02-11T10:06:37.673 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.47.5, , 00000000001.23.5, , 00000000001.43.5, , , 00000000001.55.5, , , ] 2021-02-11T10:06:37.673 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:37.668+0000 7f33166ac080 0 bucket checkpoint complete 2021-02-11T10:06:37.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:37.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:37.708 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:37.756 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:37.757 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:37.997 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:37.993+0000 7fcf6ae62080 1 bucket sync caught up with source: 2021-02-11T10:06:37.997 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.54.5, , 00000000001.23.5, , 00000000001.45.5, , , 00000000001.58.5, , , ] 2021-02-11T10:06:37.997 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.54.5, , 00000000001.23.5, , 00000000001.45.5, , , 00000000001.58.5, , , ] 2021-02-11T10:06:37.997 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:37.993+0000 7fcf6ae62080 0 bucket checkpoint complete 2021-02-11T10:06:38.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:38.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:38.031 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:38.079 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:38.079 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:38.325 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:38.321+0000 7fd3a06df080 1 bucket sync caught up with source: 2021-02-11T10:06:38.325 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.54.5, , 00000000001.23.5, , 00000000001.45.5, , , 00000000001.58.5, , , ] 2021-02-11T10:06:38.325 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.54.5, , 00000000001.23.5, , 00000000001.45.5, , , 00000000001.58.5, , , ] 2021-02-11T10:06:38.325 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:38.321+0000 7fd3a06df080 0 bucket checkpoint complete 2021-02-11T10:06:38.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:38.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:38.361 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:38.406 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:38.407 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:38.654 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:38.649+0000 7fc01f40b080 1 bucket sync caught up with source: 2021-02-11T10:06:38.654 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.62.5, , 00000000001.25.5, , 00000000001.42.5, , , 00000000001.52.5, , , ] 2021-02-11T10:06:38.654 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.62.5, , 00000000001.25.5, , 00000000001.42.5, , , 00000000001.52.5, , , ] 2021-02-11T10:06:38.654 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:38.649+0000 7fc01f40b080 0 bucket checkpoint complete 2021-02-11T10:06:38.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:38.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:38.685 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:38.732 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:38.732 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:38.963 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:38.960+0000 7f54f49e2080 1 bucket sync caught up with source: 2021-02-11T10:06:38.963 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.62.5, , 00000000001.25.5, , 00000000001.42.5, , , 00000000001.52.5, , , ] 2021-02-11T10:06:38.963 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.62.5, , 00000000001.25.5, , 00000000001.42.5, , , 00000000001.52.5, , , ] 2021-02-11T10:06:38.963 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:38.960+0000 7f54f49e2080 0 bucket checkpoint complete 2021-02-11T10:06:39.002 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-63 zones={test-zone2, test-zone1} 2021-02-11T10:06:39.661 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:06:39.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:39.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:39.669 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:39.669 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:39.669 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:06:39.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:39.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:39.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:39.676 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:39.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:39.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:39.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:39.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:40.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:40.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:40.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:40.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:40.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:40.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:40.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:40.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:40.750 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-63 zones={test-zone2, test-zone1} 2021-02-11T10:06:40.750 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-63 zones={test-zone3, test-zone1} 2021-02-11T10:06:41.425 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:06:41.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:41.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:41.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:41.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:41.430 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:06:41.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:41.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:41.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:41.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:41.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:41.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:41.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:41.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:41.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:41.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:41.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:42.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:42.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:42.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:42.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:42.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:42.497 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-63 zones={test-zone3, test-zone1} 2021-02-11T10:06:42.497 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-63 zones={test-zone3, test-zone2} 2021-02-11T10:06:42.933 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:06:42.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:42.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:42.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:42.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:42.941 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:06:42.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:42.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:42.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:42.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:42.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:43.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:43.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:43.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:43.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:43.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:43.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:43.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:43.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:43.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:43.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:43.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:44.034 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-63 zones={test-zone3, test-zone2} 2021-02-11T10:06:44.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:44.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:44.035 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:44.077 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:44.077 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:44.338 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:44.336+0000 7f51315b5080 1 bucket sync caught up with source: 2021-02-11T10:06:44.338 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.61.5, , 00000000001.55.5, , 00000000001.40.5, , , 00000000001.53.5, , , ] 2021-02-11T10:06:44.338 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.61.5, , 00000000001.55.5, , 00000000001.40.5, , , 00000000001.53.5, , , ] 2021-02-11T10:06:44.338 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:44.336+0000 7f51315b5080 0 bucket checkpoint complete 2021-02-11T10:06:44.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:44.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:44.373 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:44.421 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:44.422 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:44.655 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:44.649+0000 7fa794911080 1 bucket sync caught up with source: 2021-02-11T10:06:44.655 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.61.5, , 00000000001.55.5, , 00000000001.40.5, , , 00000000001.53.5, , , ] 2021-02-11T10:06:44.655 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.61.5, , 00000000001.55.5, , 00000000001.40.5, , , 00000000001.53.5, , , ] 2021-02-11T10:06:44.655 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:44.649+0000 7fa794911080 0 bucket checkpoint complete 2021-02-11T10:06:44.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:44.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:44.692 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:44.739 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:44.739 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:44.971 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:44.968+0000 7fb1b42a8080 1 bucket sync caught up with source: 2021-02-11T10:06:44.971 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.61.5, , 00000000001.55.5, , 00000000001.40.5, , , 00000000001.53.5, , , ] 2021-02-11T10:06:44.971 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.61.5, , 00000000001.55.5, , 00000000001.40.5, , , 00000000001.53.5, , , ] 2021-02-11T10:06:44.972 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:44.968+0000 7fb1b42a8080 0 bucket checkpoint complete 2021-02-11T10:06:45.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:45.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:45.005 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:45.053 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:45.053 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:45.309 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:45.305+0000 7f5d24396080 1 bucket sync caught up with source: 2021-02-11T10:06:45.309 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.65.5, , 00000000001.50.5, , 00000000001.36.5, , , 00000000001.52.5, , , ] 2021-02-11T10:06:45.309 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.65.5, , 00000000001.50.5, , 00000000001.36.5, , , 00000000001.52.5, , , ] 2021-02-11T10:06:45.309 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:45.305+0000 7f5d24396080 0 bucket checkpoint complete 2021-02-11T10:06:45.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:45.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:45.346 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:45.391 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:45.392 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:45.625+0000 7fd616f44080 1 bucket sync caught up with source: 2021-02-11T10:06:45.629 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.65.5, , 00000000001.50.5, , 00000000001.36.5, , , 00000000001.52.5, , , ] 2021-02-11T10:06:45.629 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.65.5, , 00000000001.50.5, , 00000000001.36.5, , , 00000000001.52.5, , , ] 2021-02-11T10:06:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:45.625+0000 7fd616f44080 0 bucket checkpoint complete 2021-02-11T10:06:45.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:45.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:45.662 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:45.751 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:45.752 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:45.993 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:45.989+0000 7f1c34219080 1 bucket sync caught up with source: 2021-02-11T10:06:45.993 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000001.62.5, , 00000000001.58.5, , 00000000001.39.5, , , 00000000001.60.5, , , ] 2021-02-11T10:06:45.993 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000001.62.5, , 00000000001.58.5, , 00000000001.39.5, , , 00000000001.60.5, , , ] 2021-02-11T10:06:45.993 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:06:45.989+0000 7f1c34219080 0 bucket checkpoint complete 2021-02-11T10:06:46.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:46.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:46.024 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:46.072 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:46.072 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:46.313 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:46.312+0000 7f5121f31080 1 bucket sync caught up with source: 2021-02-11T10:06:46.313 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.62.5, , 00000000001.58.5, , 00000000001.39.5, , , 00000000001.60.5, , , ] 2021-02-11T10:06:46.313 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.62.5, , 00000000001.58.5, , 00000000001.39.5, , , 00000000001.60.5, , , ] 2021-02-11T10:06:46.313 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:46.312+0000 7f5121f31080 0 bucket checkpoint complete 2021-02-11T10:06:46.348 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-64 zones={test-zone2, test-zone1} 2021-02-11T10:06:46.638 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:06:46.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:46.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:46.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:46.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:46.644 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:06:46.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:46.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:46.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:46.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:46.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:46.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:46.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:46.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:46.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:47.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:47.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:47.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:47.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:47.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:47.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:47.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:47.730 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-64 zones={test-zone2, test-zone1} 2021-02-11T10:06:47.730 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-64 zones={test-zone3, test-zone1} 2021-02-11T10:06:48.512 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:06:48.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:48.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:48.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:48.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:48.518 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:06:48.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:48.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:48.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:48.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:48.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:48.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:48.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:48.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:48.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:48.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:49.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:49.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:49.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:49.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:49.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:49.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:49.618 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-64 zones={test-zone3, test-zone1} 2021-02-11T10:06:49.619 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-64 zones={test-zone3, test-zone2} 2021-02-11T10:06:50.486 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:06:50.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:50.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:50.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:50.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:50.495 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:06:50.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:06:50.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:06:50.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:06:50.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:06:50.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:50.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:50.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:06:50.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:50.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:50.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:06:51.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:51.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:51.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:06:51.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:51.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:51.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:06:51.587 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-64 zones={test-zone3, test-zone2} 2021-02-11T10:06:51.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'erivkc-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:06:51.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'erivkc-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:06:51.588 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket erivkc-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:06:51.632 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:51.632 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:51.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'erivkc-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:06:51.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'erivkc-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:06:51.956 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket erivkc-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:06:51.993 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:51.993 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:52.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'erivkc-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:06:52.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'erivkc-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:06:52.318 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket erivkc-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:06:52.400 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:52.400 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:52.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:52.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:52.731 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:52.816 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:52.816 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:53.041 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:06:53.082 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038012.674132_435.1","last_update":"2021-02-11T10:06:52.674132Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:06:53.082 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:06:53.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:53.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:53.083 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:53.122 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:53.122 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:53.457 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038012.674132_435.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:06:52.674132Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:06:53.457 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:06:53.458 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:06:53.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:53.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:53.458 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:53.509 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:53.509 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:53.752 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038012.674132_435.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:06:52.674132Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:06:53.752 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:06:53.753 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:06:53.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:53.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:53.753 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:53.794 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:53.794 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:54.015 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038012.674132_435.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:06:52.674132Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:06:54.016 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:06:56.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'erivkc-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:06:56.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'erivkc-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:06:56.819 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket erivkc-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:06:56.859 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:56.859 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:57.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'erivkc-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:06:57.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'erivkc-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:06:57.180 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket erivkc-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:06:57.266 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:57.266 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:57.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'erivkc-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:06:57.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'erivkc-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:06:57.614 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket erivkc-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:06:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:58.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:58.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:06:58.104 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:06:58.148 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:58.148 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:58.426 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:06:58.424+0000 7f2c243f6080 1 waiting for incremental sync to catch up: 2021-02-11T10:06:58.427 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , , , ] 2021-02-11T10:06:58.427 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000006.64.1, 00000000004.51.1, 00000000006.83.1, 00000000004.70.1, 00000000006.65.1, 00000000004.84.1, 00000000004.47.1, 00000000006.47.1, 00000000004.164.1, 00000000004.71.1, 00000000004.55.1] 2021-02-11T10:07:03.432 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:03.428+0000 7f2c243f6080 1 waiting for incremental sync to catch up: 2021-02-11T10:07:18.575 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000006.64.1, , 00000000006.83.1, 00000000004.70.1, 00000000006.65.1, , 00000000004.47.1, 00000000006.47.1, 00000000004.164.1, , 00000000004.55.1] 2021-02-11T10:07:18.575 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000006.64.1, 00000000004.51.1, 00000000006.83.1, 00000000004.70.1, 00000000006.65.1, 00000000004.84.1, 00000000004.47.1, 00000000006.47.1, 00000000004.164.1, 00000000004.71.1, 00000000004.55.1] 2021-02-11T10:07:18.577 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:08.436+0000 7f2c243f6080 1 waiting for incremental sync to catch up: 2021-02-11T10:07:18.577 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000006.64.1, , 00000000006.83.1, 00000000004.70.1, 00000000006.65.1, , 00000000004.47.1, 00000000006.47.1, 00000000004.164.1, , 00000000004.55.1] 2021-02-11T10:07:18.577 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000006.64.1, 00000000004.51.1, 00000000006.83.1, 00000000004.70.1, 00000000006.65.1, 00000000004.84.1, 00000000004.47.1, 00000000006.47.1, 00000000004.164.1, 00000000004.71.1, 00000000004.55.1] 2021-02-11T10:07:18.577 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:13.440+0000 7f2c243f6080 1 waiting for incremental sync to catch up: 2021-02-11T10:07:18.577 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000006.64.1, , 00000000006.83.1, 00000000004.70.1, 00000000006.65.1, , 00000000004.47.1, 00000000006.47.1, 00000000004.164.1, , 00000000004.55.1] 2021-02-11T10:07:18.577 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000006.64.1, 00000000004.51.1, 00000000006.83.1, 00000000004.70.1, 00000000006.65.1, 00000000004.84.1, 00000000004.47.1, 00000000006.47.1, 00000000004.164.1, 00000000004.71.1, 00000000004.55.1] 2021-02-11T10:07:18.577 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:18.444+0000 7f2c243f6080 1 bucket sync caught up with source: 2021-02-11T10:07:18.577 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000006.64.1, 00000000004.51.1, 00000000006.83.1, 00000000004.70.1, 00000000006.65.1, 00000000004.84.1, 00000000004.47.1, 00000000006.47.1, 00000000004.164.1, 00000000004.71.1, 00000000004.55.1] 2021-02-11T10:07:18.577 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000006.64.1, 00000000004.51.1, 00000000006.83.1, 00000000004.70.1, 00000000006.65.1, 00000000004.84.1, 00000000004.47.1, 00000000006.47.1, 00000000004.164.1, 00000000004.71.1, 00000000004.55.1] 2021-02-11T10:07:18.577 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:18.444+0000 7f2c243f6080 0 bucket checkpoint complete 2021-02-11T10:07:18.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:18.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:18.578 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:18.630 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:18.630 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:18.836 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:18.833+0000 7f5dcef57080 1 bucket sync caught up with source: 2021-02-11T10:07:18.836 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000006.64.1, 00000000004.51.1, 00000000006.83.1, 00000000004.70.1, 00000000006.65.1, 00000000004.84.1, 00000000004.47.1, 00000000006.47.1, 00000000004.164.1, 00000000004.71.1, 00000000004.55.1] 2021-02-11T10:07:18.836 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000006.64.1, 00000000004.51.1, 00000000006.83.1, 00000000004.70.1, 00000000006.65.1, 00000000004.84.1, 00000000004.47.1, 00000000006.47.1, 00000000004.164.1, 00000000004.71.1, 00000000004.55.1] 2021-02-11T10:07:18.836 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:18.833+0000 7f5dcef57080 0 bucket checkpoint complete 2021-02-11T10:07:18.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:18.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:18.859 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:18.908 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:18.908 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:19.147 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:19.144+0000 7f43593f0080 1 bucket sync caught up with source: 2021-02-11T10:07:19.148 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000006.64.1, 00000000004.51.1, 00000000006.83.1, 00000000004.70.1, 00000000006.65.1, 00000000004.84.1, 00000000004.47.1, 00000000006.47.1, 00000000004.164.1, 00000000004.71.1, 00000000004.55.1] 2021-02-11T10:07:19.148 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000006.64.1, 00000000004.51.1, 00000000006.83.1, 00000000004.70.1, 00000000006.65.1, 00000000004.84.1, 00000000004.47.1, 00000000006.47.1, 00000000004.164.1, 00000000004.71.1, 00000000004.55.1] 2021-02-11T10:07:19.148 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:19.144+0000 7f43593f0080 0 bucket checkpoint complete 2021-02-11T10:07:19.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:19.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:19.180 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:19.230 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:19.461 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:19.457+0000 7f03dc7f6080 1 bucket sync caught up with source: 2021-02-11T10:07:19.461 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000004.65.5, 00000000002.56.1, 00000000004.92.5, 00000000002.73.1, 00000000004.66.5, 00000000002.66.1, 00000000002.57.1, 00000000004.42.5, 00000000002.159.1, 00000000002.75.1, 00000000002.50.1] 2021-02-11T10:07:19.461 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000004.65.5, 00000000002.56.1, 00000000004.92.5, 00000000002.73.1, 00000000004.66.5, 00000000002.66.1, 00000000002.57.1, 00000000004.42.5, 00000000002.159.1, 00000000002.75.1, 00000000002.50.1] 2021-02-11T10:07:19.461 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:19.457+0000 7f03dc7f6080 0 bucket checkpoint complete 2021-02-11T10:07:19.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:19.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:19.498 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:19.544 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:19.544 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:19.797 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:19.793+0000 7f8af3162080 1 bucket sync caught up with source: 2021-02-11T10:07:19.797 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000004.65.5, 00000000002.56.1, 00000000004.92.5, 00000000002.73.1, 00000000004.66.5, 00000000002.66.1, 00000000002.57.1, 00000000004.42.5, 00000000002.159.1, 00000000002.75.1, 00000000002.50.1] 2021-02-11T10:07:19.797 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000004.65.5, 00000000002.56.1, 00000000004.92.5, 00000000002.73.1, 00000000004.66.5, 00000000002.66.1, 00000000002.57.1, 00000000004.42.5, 00000000002.159.1, 00000000002.75.1, 00000000002.50.1] 2021-02-11T10:07:19.797 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:19.793+0000 7f8af3162080 0 bucket checkpoint complete 2021-02-11T10:07:19.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:19.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:19.828 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:19.871 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:19.871 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:20.146 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:20.141+0000 7f7f4c33b080 1 bucket sync caught up with source: 2021-02-11T10:07:20.146 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000005.65.5, 00000000002.51.1, 00000000004.101.5, 00000000002.71.1, 00000000004.61.5, 00000000002.66.1, 00000000002.48.1, 00000000004.47.5, 00000000002.175.1, 00000000002.72.1, 00000000002.48.1] 2021-02-11T10:07:20.146 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000005.65.5, 00000000002.51.1, 00000000004.101.5, 00000000002.71.1, 00000000004.61.5, 00000000002.66.1, 00000000002.48.1, 00000000004.47.5, 00000000002.175.1, 00000000002.72.1, 00000000002.48.1] 2021-02-11T10:07:20.146 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:20.141+0000 7f7f4c33b080 0 bucket checkpoint complete 2021-02-11T10:07:20.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:20.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:20.178 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:20.225 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:20.225 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:20.469 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:20.468+0000 7f757855e080 1 bucket sync caught up with source: 2021-02-11T10:07:20.469 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000005.65.5, 00000000002.51.1, 00000000004.101.5, 00000000002.71.1, 00000000004.61.5, 00000000002.66.1, 00000000002.48.1, 00000000004.47.5, 00000000002.175.1, 00000000002.72.1, 00000000002.48.1] 2021-02-11T10:07:20.469 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000005.65.5, 00000000002.51.1, 00000000004.101.5, 00000000002.71.1, 00000000004.61.5, 00000000002.66.1, 00000000002.48.1, 00000000004.47.5, 00000000002.175.1, 00000000002.72.1, 00000000002.48.1] 2021-02-11T10:07:20.469 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:20.468+0000 7f757855e080 0 bucket checkpoint complete 2021-02-11T10:07:20.508 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-62 zones={test-zone2, test-zone1} 2021-02-11T10:07:20.659 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:07:20.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:07:20.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:07:20.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:07:20.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:07:20.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:07:20.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:07:20.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:07:20.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:07:20.667 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:07:20.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:07:20.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:07:20.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:07:20.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:07:20.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:07:20.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:07:20.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:07:20.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:07:20.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:20.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:20.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:20.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:21.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:21.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:21.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:21.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:21.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:21.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:21.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:21.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:21.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:21.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:21.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:22.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:22.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:22.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:22.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:22.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:22.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:22.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:22.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:22.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:22.853 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-62 zones={test-zone2, test-zone1} 2021-02-11T10:07:22.853 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-62 zones={test-zone3, test-zone1} 2021-02-11T10:07:23.334 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:07:23.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:07:23.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:07:23.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:07:23.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:07:23.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:07:23.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:07:23.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:07:23.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:07:23.340 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:07:23.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:07:23.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:07:23.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:07:23.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:07:23.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:07:23.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:07:23.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:07:23.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:07:23.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:23.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:23.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:23.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:23.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:23.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:23.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:23.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:24.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:24.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:24.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:24.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:24.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:24.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:24.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:24.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:24.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:24.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:25.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:25.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:25.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:25.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:25.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:25.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:25.533 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-62 zones={test-zone3, test-zone1} 2021-02-11T10:07:25.533 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-62 zones={test-zone3, test-zone2} 2021-02-11T10:07:26.353 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:07:26.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:07:26.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:07:26.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:07:26.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:07:26.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:07:26.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:07:26.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:07:26.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:07:26.361 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:07:26.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:07:26.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:07:26.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:07:26.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:07:26.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:07:26.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:07:26.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:07:26.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:07:26.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:26.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:26.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:26.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:26.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:26.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:26.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:26.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:27.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:27.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:27.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:27.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:27.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:27.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:27.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:27.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:27.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:27.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:27.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:28.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:28.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:28.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:28.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:28.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:28.489 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-62 zones={test-zone3, test-zone2} 2021-02-11T10:07:28.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:28.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:28.490 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:28.534 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:28.534 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:28.779 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:28.775+0000 7f14a5dbd080 1 bucket sync caught up with source: 2021-02-11T10:07:28.780 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000006.53.5, 00000000004.66.1, 00000000006.29.5, 00000000004.48.1, 00000000006.57.5, 00000000004.39.1, 00000000004.86.1, 00000000006.61.5, 00000000004.71.1, 00000000004.67.1, 00000000004.55.1] 2021-02-11T10:07:28.780 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000006.53.5, 00000000004.66.1, 00000000006.29.5, 00000000004.48.1, 00000000006.57.5, 00000000004.39.1, 00000000004.86.1, 00000000006.61.5, 00000000004.71.1, 00000000004.67.1, 00000000004.55.1] 2021-02-11T10:07:28.780 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:28.775+0000 7f14a5dbd080 0 bucket checkpoint complete 2021-02-11T10:07:28.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:28.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:28.816 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:28.862 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:28.862 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:29.115 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:29.108+0000 7f402742f080 1 bucket sync caught up with source: 2021-02-11T10:07:29.115 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000006.53.5, 00000000004.66.1, 00000000006.29.5, 00000000004.48.1, 00000000006.57.5, 00000000004.39.1, 00000000004.86.1, 00000000006.61.5, 00000000004.71.1, 00000000004.67.1, 00000000004.55.1] 2021-02-11T10:07:29.115 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000006.53.5, 00000000004.66.1, 00000000006.29.5, 00000000004.48.1, 00000000006.57.5, 00000000004.39.1, 00000000004.86.1, 00000000006.61.5, 00000000004.71.1, 00000000004.67.1, 00000000004.55.1] 2021-02-11T10:07:29.115 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:29.108+0000 7f402742f080 0 bucket checkpoint complete 2021-02-11T10:07:29.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:29.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:29.145 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:29.192 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:29.193 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:29.415 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:29.411+0000 7fc7ac4f4080 1 bucket sync caught up with source: 2021-02-11T10:07:29.415 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000006.53.5, 00000000004.66.1, 00000000006.29.5, 00000000004.48.1, 00000000006.57.5, 00000000004.39.1, 00000000004.86.1, 00000000006.61.5, 00000000004.71.1, 00000000004.67.1, 00000000004.55.1] 2021-02-11T10:07:29.415 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000006.53.5, 00000000004.66.1, 00000000006.29.5, 00000000004.48.1, 00000000006.57.5, 00000000004.39.1, 00000000004.86.1, 00000000006.61.5, 00000000004.71.1, 00000000004.67.1, 00000000004.55.1] 2021-02-11T10:07:29.415 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:29.411+0000 7fc7ac4f4080 0 bucket checkpoint complete 2021-02-11T10:07:29.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:29.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:29.452 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:29.501 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:29.753 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:29.748+0000 7f7cdcf69080 1 bucket sync caught up with source: 2021-02-11T10:07:29.753 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000004.59.1, 00000000002.59.1, 00000000004.28.1, 00000000002.45.1, 00000000004.54.1, 00000000002.36.1, 00000000002.99.1, 00000000004.63.1, 00000000002.102.1, 00000000002.60.1, 00000000002.55.1] 2021-02-11T10:07:29.753 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000004.59.1, 00000000002.59.1, 00000000004.28.1, 00000000002.45.1, 00000000004.54.1, 00000000002.36.1, 00000000002.99.1, 00000000004.63.1, 00000000002.102.1, 00000000002.60.1, 00000000002.55.1] 2021-02-11T10:07:29.753 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:29.748+0000 7f7cdcf69080 0 bucket checkpoint complete 2021-02-11T10:07:29.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:29.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:29.788 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:29.835 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:29.835 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:30.068+0000 7f5976171080 1 bucket sync caught up with source: 2021-02-11T10:07:30.077 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000004.59.1, 00000000002.59.1, 00000000004.28.1, 00000000002.45.1, 00000000004.54.1, 00000000002.36.1, 00000000002.99.1, 00000000004.63.1, 00000000002.102.1, 00000000002.60.1, 00000000002.55.1] 2021-02-11T10:07:30.077 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000004.59.1, 00000000002.59.1, 00000000004.28.1, 00000000002.45.1, 00000000004.54.1, 00000000002.36.1, 00000000002.99.1, 00000000004.63.1, 00000000002.102.1, 00000000002.60.1, 00000000002.55.1] 2021-02-11T10:07:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:30.068+0000 7f5976171080 0 bucket checkpoint complete 2021-02-11T10:07:30.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:30.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:30.123 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:30.168 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:30.168 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:30.394 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:30.388+0000 7f5329f72080 1 bucket sync caught up with source: 2021-02-11T10:07:30.394 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000004.68.5, 00000000002.55.1, 00000000004.31.5, 00000000002.48.1, 00000000004.56.5, 00000000002.36.1, 00000000002.103.1, 00000000004.58.5, 00000000002.80.1, 00000000002.53.1, 00000000002.50.1] 2021-02-11T10:07:30.395 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000004.68.5, 00000000002.55.1, 00000000004.31.5, 00000000002.48.1, 00000000004.56.5, 00000000002.36.1, 00000000002.103.1, 00000000004.58.5, 00000000002.80.1, 00000000002.53.1, 00000000002.50.1] 2021-02-11T10:07:30.395 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:30.388+0000 7f5329f72080 0 bucket checkpoint complete 2021-02-11T10:07:30.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:30.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:30.426 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:30.474 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:30.474 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:30.712 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:30.707+0000 7fb5b0108080 1 bucket sync caught up with source: 2021-02-11T10:07:30.712 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000004.68.5, 00000000002.55.1, 00000000004.31.5, 00000000002.48.1, 00000000004.56.5, 00000000002.36.1, 00000000002.103.1, 00000000004.58.5, 00000000002.80.1, 00000000002.53.1, 00000000002.50.1] 2021-02-11T10:07:30.712 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000004.68.5, 00000000002.55.1, 00000000004.31.5, 00000000002.48.1, 00000000004.56.5, 00000000002.36.1, 00000000002.103.1, 00000000004.58.5, 00000000002.80.1, 00000000002.53.1, 00000000002.50.1] 2021-02-11T10:07:30.712 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:30.707+0000 7fb5b0108080 0 bucket checkpoint complete 2021-02-11T10:07:30.748 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-63 zones={test-zone2, test-zone1} 2021-02-11T10:07:31.812 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:07:31.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:07:31.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:07:31.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:07:31.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:07:31.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:07:31.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:07:31.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:07:31.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:07:31.823 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:07:31.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:07:31.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:07:31.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:07:31.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:07:31.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:07:31.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:07:31.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:07:31.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:07:31.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:31.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:32.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:32.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:32.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:32.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:32.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:32.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:32.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:32.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:32.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:32.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:32.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:32.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:33.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:33.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:33.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:33.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:33.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:33.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:33.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:33.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:33.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:33.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:34.009 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-63 zones={test-zone2, test-zone1} 2021-02-11T10:07:34.009 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-63 zones={test-zone3, test-zone1} 2021-02-11T10:07:34.538 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:07:34.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:07:34.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:07:34.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:07:34.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:07:34.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:07:34.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:07:34.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:07:34.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:07:34.544 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:07:34.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:07:34.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:07:34.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:07:34.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:07:34.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:07:34.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:07:34.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:07:34.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:07:34.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:34.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:34.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:34.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:34.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:34.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:35.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:35.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:35.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:35.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:35.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:35.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:35.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:35.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:35.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:35.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:35.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:36.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:36.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:36.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:36.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:36.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:36.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:36.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:36.708 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-63 zones={test-zone3, test-zone1} 2021-02-11T10:07:47.635 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-63 zones={test-zone3, test-zone2} 2021-02-11T10:07:49.184 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:07:49.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:07:49.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:07:49.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:07:49.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:07:49.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:07:49.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:07:49.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:07:49.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:07:49.193 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:07:49.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:07:49.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:07:49.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:07:49.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:07:49.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:07:49.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:07:49.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:07:49.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:07:49.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:49.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:49.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:49.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:49.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:49.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:49.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:49.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:49.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:50.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:50.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:50.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:50.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:50.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:50.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:50.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:50.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:50.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:50.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:50.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:50.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:51.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:51.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:51.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:51.361 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-63 zones={test-zone3, test-zone2} 2021-02-11T10:07:51.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:51.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:51.362 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:51.407 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:51.408 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:51.653 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:51.651+0000 7f55cbbe1080 1 bucket sync caught up with source: 2021-02-11T10:07:51.654 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000006.75.5, 00000000004.48.1, 00000000006.70.5, 00000000004.51.1, 00000000006.54.5, 00000000004.69.1, 00000000004.66.1, 00000000006.71.5, 00000000004.68.1, 00000000004.55.1, 00000000004.52.1] 2021-02-11T10:07:51.654 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000006.75.5, 00000000004.48.1, 00000000006.70.5, 00000000004.51.1, 00000000006.54.5, 00000000004.69.1, 00000000004.66.1, 00000000006.71.5, 00000000004.68.1, 00000000004.55.1, 00000000004.52.1] 2021-02-11T10:07:51.654 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:51.651+0000 7f55cbbe1080 0 bucket checkpoint complete 2021-02-11T10:07:51.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:51.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:51.693 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:51.740 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:51.740 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:51.970 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:51.964+0000 7f5abd283080 1 bucket sync caught up with source: 2021-02-11T10:07:51.970 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000006.75.5, 00000000004.48.1, 00000000006.70.5, 00000000004.51.1, 00000000006.54.5, 00000000004.69.1, 00000000004.66.1, 00000000006.71.5, 00000000004.68.1, 00000000004.55.1, 00000000004.52.1] 2021-02-11T10:07:51.970 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000006.75.5, 00000000004.48.1, 00000000006.70.5, 00000000004.51.1, 00000000006.54.5, 00000000004.69.1, 00000000004.66.1, 00000000006.71.5, 00000000004.68.1, 00000000004.55.1, 00000000004.52.1] 2021-02-11T10:07:51.970 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:51.964+0000 7f5abd283080 0 bucket checkpoint complete 2021-02-11T10:07:52.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:52.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:52.005 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:52.053 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:52.053 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:52.287 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:52.283+0000 7f6980834080 1 bucket sync caught up with source: 2021-02-11T10:07:52.287 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000006.75.5, 00000000004.48.1, 00000000006.70.5, 00000000004.51.1, 00000000006.54.5, 00000000004.69.1, 00000000004.66.1, 00000000006.71.5, 00000000004.68.1, 00000000004.55.1, 00000000004.52.1] 2021-02-11T10:07:52.287 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000006.75.5, 00000000004.48.1, 00000000006.70.5, 00000000004.51.1, 00000000006.54.5, 00000000004.69.1, 00000000004.66.1, 00000000006.71.5, 00000000004.68.1, 00000000004.55.1, 00000000004.52.1] 2021-02-11T10:07:52.287 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:52.283+0000 7f6980834080 0 bucket checkpoint complete 2021-02-11T10:07:52.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:52.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:52.320 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:52.368 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:52.369 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:52.632 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:52.628+0000 7f42d0452080 1 bucket sync caught up with source: 2021-02-11T10:07:52.633 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000004.79.5, 00000000002.55.1, 00000000004.68.5, 00000000002.65.1, 00000000004.50.5, 00000000002.56.1, 00000000002.60.1, 00000000004.67.5, 00000000002.55.1, 00000000002.54.1, 00000000002.44.1] 2021-02-11T10:07:52.633 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000004.79.5, 00000000002.55.1, 00000000004.68.5, 00000000002.65.1, 00000000004.50.5, 00000000002.56.1, 00000000002.60.1, 00000000004.67.5, 00000000002.55.1, 00000000002.54.1, 00000000002.44.1] 2021-02-11T10:07:52.633 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:52.628+0000 7f42d0452080 0 bucket checkpoint complete 2021-02-11T10:07:52.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:52.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:52.671 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:52.715 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:52.715 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:52.941 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:52.936+0000 7fb3d7179080 1 bucket sync caught up with source: 2021-02-11T10:07:52.942 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000004.79.5, 00000000002.55.1, 00000000004.68.5, 00000000002.65.1, 00000000004.50.5, 00000000002.56.1, 00000000002.60.1, 00000000004.67.5, 00000000002.55.1, 00000000002.54.1, 00000000002.44.1] 2021-02-11T10:07:52.942 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000004.79.5, 00000000002.55.1, 00000000004.68.5, 00000000002.65.1, 00000000004.50.5, 00000000002.56.1, 00000000002.60.1, 00000000004.67.5, 00000000002.55.1, 00000000002.54.1, 00000000002.44.1] 2021-02-11T10:07:52.942 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:52.936+0000 7fb3d7179080 0 bucket checkpoint complete 2021-02-11T10:07:52.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:52.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:52.979 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:53.068 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:53.068 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:53.316 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:53.308+0000 7f761ce70080 1 bucket sync caught up with source: 2021-02-11T10:07:53.316 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [00000000004.73.1, 00000000002.55.1, 00000000004.71.1, 00000000002.62.1, 00000000004.49.1, 00000000002.62.1, 00000000002.72.1, 00000000004.73.1, 00000000002.66.1, 00000000002.56.1, 00000000002.51.1] 2021-02-11T10:07:53.316 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [00000000004.73.1, 00000000002.55.1, 00000000004.71.1, 00000000002.62.1, 00000000004.49.1, 00000000002.62.1, 00000000002.72.1, 00000000004.73.1, 00000000002.66.1, 00000000002.56.1, 00000000002.51.1] 2021-02-11T10:07:53.316 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:07:53.308+0000 7f761ce70080 0 bucket checkpoint complete 2021-02-11T10:07:53.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:53.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:07:53.349 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:07:53.399 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:53.400 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:53.627 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:53.623+0000 7f5021c3c080 1 bucket sync caught up with source: 2021-02-11T10:07:53.627 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000004.73.1, 00000000002.55.1, 00000000004.71.1, 00000000002.62.1, 00000000004.49.1, 00000000002.62.1, 00000000002.72.1, 00000000004.73.1, 00000000002.66.1, 00000000002.56.1, 00000000002.51.1] 2021-02-11T10:07:53.627 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000004.73.1, 00000000002.55.1, 00000000004.71.1, 00000000002.62.1, 00000000004.49.1, 00000000002.62.1, 00000000002.72.1, 00000000004.73.1, 00000000002.66.1, 00000000002.56.1, 00000000002.51.1] 2021-02-11T10:07:53.628 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:07:53.623+0000 7f5021c3c080 0 bucket checkpoint complete 2021-02-11T10:07:53.662 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-64 zones={test-zone2, test-zone1} 2021-02-11T10:07:54.447 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:07:54.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:07:54.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:07:54.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:07:54.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:07:54.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:07:54.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:07:54.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:07:54.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:07:54.456 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:07:54.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:07:54.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:07:54.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:07:54.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:07:54.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:07:54.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:07:54.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:07:54.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:07:54.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:54.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:54.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:54.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:54.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:54.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:55.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:55.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:55.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:55.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:55.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:55.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:07:55.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:55.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:55.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:07:55.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:55.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:56.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:07:56.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:56.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:56.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:07:56.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:56.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:56.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:07:56.670 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-64 zones={test-zone2, test-zone1} 2021-02-11T10:07:56.670 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-64 zones={test-zone3, test-zone1} 2021-02-11T10:07:58.677 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:07:59.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:07:59.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:07:59.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:07:59.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:07:59.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:07:59.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:07:59.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:07:59.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:07:59.204 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:07:59.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:07:59.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:07:59.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:07:59.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:07:59.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:07:59.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:07:59.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:07:59.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:07:59.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:59.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:59.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:07:59.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:59.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:59.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:07:59.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:59.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:07:59.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:08:00.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:08:00.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:08:00.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:08:00.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:08:00.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:08:00.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:08:00.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:08:00.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:08:00.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:08:00.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:08:00.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:08:01.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:08:01.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:08:01.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:08:01.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:08:01.430 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-64 zones={test-zone3, test-zone1} 2021-02-11T10:08:01.430 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-64 zones={test-zone3, test-zone2} 2021-02-11T10:08:01.743 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:08:01.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:08:01.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:08:01.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:08:01.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:08:01.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:08:01.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:08:01.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:08:01.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:08:01.750 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:08:01.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:08:01.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:08:01.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:08:01.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:08:01.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:08:01.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:08:01.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:08:01.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:08:01.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:08:01.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:08:01.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:08:02.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:08:02.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:08:02.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:08:02.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:08:02.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:08:02.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:08:02.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:08:02.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:08:02.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:08:02.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:08:03.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:08:03.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:08:03.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:08:03.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:08:03.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:08:03.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:08:03.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:08:03.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:08:03.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:08:03.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:08:03.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:08:04.034 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-64 zones={test-zone3, test-zone2} 2021-02-11T10:08:04.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:04.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:04.035 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:04.076 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:04.076 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:04.348 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613038036.244251_763.2","last_update":"2021-02-11T10:07:16.244251Z"},{"marker":"1_1613038018.055151_592.3","last_update":"2021-02-11T10:06:58.055151Z"},{"marker":"1_1613038036.244251_323.1","last_update":"2021-02-11T10:07:16.244251Z"},{"marker":"1_1613038036.244251_471.1","last_update":"2021-02-11T10:07:16.244251Z"}] 2021-02-11T10:08:04.348 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:08:04.349 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:04.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:04.349 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:04.393 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:04.393 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:04.644 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038036.244251_763.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:16.244251Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038018.055151_592.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:06:58.055151Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038036.244251_323.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:16.244251Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613038036.244251_471.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:16.244251Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:08:04.644 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:08:04.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:04.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:04.645 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:04.684 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:04.685 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:04.952 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613038036.244251_763.2","last_update":"2021-02-11T10:07:16.244251Z"},{"marker":"1_1613038018.055151_592.3","last_update":"2021-02-11T10:06:58.055151Z"},{"marker":"1_1613038036.244251_323.1","last_update":"2021-02-11T10:07:16.244251Z"},{"marker":"1_1613038036.244251_471.1","last_update":"2021-02-11T10:07:16.244251Z"}] 2021-02-11T10:08:04.953 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T10:08:04.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:04.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:04.953 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:05.039 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:05.311 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7045610624405686142},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038036.244251_763.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:16.244251Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038018.055151_592.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:06:58.055151Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038036.244251_323.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:16.244251Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613038036.244251_471.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:16.244251Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:08:05.312 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T10:08:05.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:05.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:05.312 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:05.396 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:05.396 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:05.672 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613038036.244251_763.2","last_update":"2021-02-11T10:07:16.244251Z"},{"marker":"1_1613038018.055151_592.3","last_update":"2021-02-11T10:06:58.055151Z"},{"marker":"1_1613038036.244251_323.1","last_update":"2021-02-11T10:07:16.244251Z"},{"marker":"1_1613038036.244251_471.1","last_update":"2021-02-11T10:07:16.244251Z"}] 2021-02-11T10:08:05.672 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T10:08:05.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:05.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:05.672 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:05.716 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:05.716 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:05.985 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1531294025241915737},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038036.244251_763.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:16.244251Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038018.055151_592.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:06:58.055151Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038036.244251_323.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:16.244251Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613038036.244251_471.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:16.244251Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:08:05.985 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T10:08:05.985 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:05.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:05.985 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:06.025 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:06.025 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:06.288 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1613038037.625962_691.9","last_update":"2021-02-11T10:07:17.625962Z"},{"marker":"1_1613038037.625962_555.9","last_update":"2021-02-11T10:07:17.625962Z"},{"marker":"1_1613038037.625962_300.9","last_update":"2021-02-11T10:07:17.625962Z"},{"marker":"1_1613038037.625962_637.6","last_update":"2021-02-11T10:07:17.625962Z"}] 2021-02-11T10:08:06.288 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T10:08:06.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:06.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:06.289 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:06.331 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:06.331 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:06.612 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3492294689851219663},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038037.625962_691.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:17.625962Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038037.625962_555.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:17.625962Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038037.625962_300.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:17.625962Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613038037.625962_637.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:17.625962Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:08:06.613 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T10:08:06.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:06.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:06.613 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:06.648 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:06.648 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:06.899 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1613038037.625962_691.9","last_update":"2021-02-11T10:07:17.625962Z"},{"marker":"1_1613038037.625962_555.9","last_update":"2021-02-11T10:07:17.625962Z"},{"marker":"1_1613038037.625962_300.9","last_update":"2021-02-11T10:07:17.625962Z"},{"marker":"1_1613038037.625962_637.6","last_update":"2021-02-11T10:07:17.625962Z"}] 2021-02-11T10:08:06.899 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T10:08:06.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:06.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:06.900 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:06.944 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:06.944 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:07.198 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7442380677805818242},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038037.625962_691.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:17.625962Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038037.625962_555.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:17.625962Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038037.625962_300.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:17.625962Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613038037.625962_637.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:17.625962Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:08:07.198 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T10:08:07.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:07.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:07.199 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:07.238 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:07.238 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:07.521 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613038036.312443_661.9","last_update":"2021-02-11T10:07:16.312443Z"},{"marker":"1_1613038036.312443_365.9","last_update":"2021-02-11T10:07:16.312443Z"},{"marker":"1_1613038036.312443_206.9","last_update":"2021-02-11T10:07:16.312443Z"},{"marker":"1_1613038036.312443_436.6","last_update":"2021-02-11T10:07:16.312443Z"}] 2021-02-11T10:08:07.521 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T10:08:07.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:07.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:07.522 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:07.610 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:07.610 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:07.892 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14314976040552073418},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038036.312443_661.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:16.312443Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038036.312443_365.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:16.312443Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038036.312443_206.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:16.312443Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613038036.312443_436.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:16.312443Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:08:16.169 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T10:08:16.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:16.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:16.169 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:16.211 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:16.211 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:16.469 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613038036.312443_661.9","last_update":"2021-02-11T10:07:16.312443Z"},{"marker":"1_1613038036.312443_365.9","last_update":"2021-02-11T10:07:16.312443Z"},{"marker":"1_1613038036.312443_206.9","last_update":"2021-02-11T10:07:16.312443Z"},{"marker":"1_1613038036.312443_436.6","last_update":"2021-02-11T10:07:16.312443Z"}] 2021-02-11T10:08:16.469 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T10:08:16.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:16.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:16.470 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:16.512 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:16.512 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:16.780 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12589397483957797345},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038036.312443_661.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:16.312443Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038036.312443_365.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:16.312443Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038036.312443_206.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:16.312443Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613038036.312443_436.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:07:16.312443Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:08:16.780 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T10:08:16.781 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2021-02-11T10:08:16.781 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-65 2021-02-11T10:08:17.317 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-66 2021-02-11T10:08:17.819 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-67 2021-02-11T10:08:19.627 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:19.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:19.627 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:19.677 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:19.677 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:19.885 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:19.887+0000 7f7f8ff0f080 1 bucket sync caught up with source: 2021-02-11T10:08:19.885 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , 00000000011.56.5, , , , , ] 2021-02-11T10:08:19.886 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , 00000000011.56.5, , , , , ] 2021-02-11T10:08:19.886 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:19.887+0000 7f7f8ff0f080 0 bucket checkpoint complete 2021-02-11T10:08:19.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:19.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:19.922 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:19.963 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:19.964 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:20.218 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:20.212+0000 7f7c64bf3080 1 bucket sync caught up with source: 2021-02-11T10:08:20.218 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000011.56.5, , , , , ] 2021-02-11T10:08:20.219 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000011.56.5, , , , , ] 2021-02-11T10:08:20.219 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:20.212+0000 7f7c64bf3080 0 bucket checkpoint complete 2021-02-11T10:08:20.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:20.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:20.257 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:20.305 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:20.305 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:20.542 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:20.543+0000 7f0e31f96080 1 bucket sync caught up with source: 2021-02-11T10:08:20.542 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , 00000000011.56.5, , , , , ] 2021-02-11T10:08:20.542 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , 00000000011.56.5, , , , , ] 2021-02-11T10:08:20.543 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:20.543+0000 7f0e31f96080 0 bucket checkpoint complete 2021-02-11T10:08:20.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:20.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:20.577 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:20.625 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:20.625 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:20.881 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:20.876+0000 7f3ccee2c080 1 bucket sync caught up with source: 2021-02-11T10:08:20.881 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000001.42.5, , , , , ] 2021-02-11T10:08:20.881 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000001.42.5, , , , , ] 2021-02-11T10:08:20.881 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:20.876+0000 7f3ccee2c080 0 bucket checkpoint complete 2021-02-11T10:08:20.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:20.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:20.920 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:20.967 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:20.967 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:21.215 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:21.208+0000 7f4af0eff080 1 bucket sync caught up with source: 2021-02-11T10:08:21.216 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000001.42.5, , , , , ] 2021-02-11T10:08:21.216 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000001.42.5, , , , , ] 2021-02-11T10:08:21.216 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:21.208+0000 7f4af0eff080 0 bucket checkpoint complete 2021-02-11T10:08:21.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:21.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:21.254 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:21.298 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:21.298 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:21.530 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:21.524+0000 7ff04c108080 1 bucket sync caught up with source: 2021-02-11T10:08:21.531 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , 00000000001.47.5, , , , , ] 2021-02-11T10:08:21.531 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , 00000000001.47.5, , , , , ] 2021-02-11T10:08:21.531 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:21.524+0000 7ff04c108080 0 bucket checkpoint complete 2021-02-11T10:08:21.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:21.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:21.564 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:21.611 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:21.611 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:21.845 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:21.847+0000 7fcb77696080 1 bucket sync caught up with source: 2021-02-11T10:08:21.845 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , 00000000001.47.5, , , , , ] 2021-02-11T10:08:21.845 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , 00000000001.47.5, , , , , ] 2021-02-11T10:08:21.845 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:21.847+0000 7fcb77696080 0 bucket checkpoint complete 2021-02-11T10:08:21.880 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-65 zones={test-zone2, test-zone1} 2021-02-11T10:08:22.871 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:08:22.919 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-02-11T10:08:22.919 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:08:22.925 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-02-11T10:08:22.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T10:08:23.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T10:08:23.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T10:08:23.762 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-65 zones={test-zone2, test-zone1} 2021-02-11T10:08:23.763 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-65 zones={test-zone3, test-zone1} 2021-02-11T10:08:24.322 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:08:24.367 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-02-11T10:08:24.367 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:08:24.374 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-02-11T10:08:24.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T10:08:24.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T10:08:24.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T10:08:25.270 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-65 zones={test-zone3, test-zone1} 2021-02-11T10:08:25.271 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-65 zones={test-zone3, test-zone2} 2021-02-11T10:08:25.598 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:08:25.605 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-02-11T10:08:25.605 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:08:25.651 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-02-11T10:08:25.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T10:08:25.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T10:08:26.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T10:08:26.523 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-65 zones={test-zone3, test-zone2} 2021-02-11T10:08:26.523 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multipart_object_sync ... ok 2021-02-11T10:08:26.523 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-68 2021-02-11T10:08:26.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:26.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:26.903 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:26.943 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:26.943 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:26.987 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2021-02-11T10:08:26.987+0000 7f855affd700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:08:26.987 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2021-02-11T10:08:26.987+0000 7f85717fa700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:08:26.988 INFO:tasks.rgw.c1.client.1.smithi026.stdout:2021-02-11T10:08:26.980+0000 7fd64c7f8700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:08:26.988 INFO:tasks.rgw.c1.client.1.smithi026.stdout:2021-02-11T10:08:26.980+0000 7fd6467ec700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:08:27.166 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:08:27.201 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038106.809686_488.1","last_update":"2021-02-11T10:08:26.809686Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:08:27.202 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:08:27.202 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:27.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:27.202 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:27.241 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:27.241 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:27.488 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038106.809686_488.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:26.809686Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:08:27.488 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:08:27.488 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:08:27.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:27.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:27.489 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:27.529 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:27.530 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:27.788 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038106.809686_488.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:26.809686Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:08:27.788 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:08:27.788 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:08:27.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:27.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:27.789 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:27.830 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:27.830 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:28.101 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038106.809686_488.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:26.809686Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:08:28.101 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:08:28.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:28.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:28.102 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-68 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:28.196 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:28.196 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:28.443 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:28.443+0000 7f73ce164080 1 bucket sync caught up with source: 2021-02-11T10:08:28.443 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000001.30.5, , , , 00000000001.34.5, , , , , ] 2021-02-11T10:08:28.444 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000001.30.5, , , , 00000000001.34.5, , , , , ] 2021-02-11T10:08:28.444 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:28.443+0000 7f73ce164080 0 bucket checkpoint complete 2021-02-11T10:08:28.814 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_encrypted_object_sync ... ok 2021-02-11T10:08:28.815 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-69 2021-02-11T10:08:29.716 INFO:tasks.rgw.c1.client.1.smithi026.stdout:2021-02-11T10:08:29.711+0000 7fd657fff700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:08:29.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:29.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:29.786 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:29.790 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2021-02-11T10:08:29.791+0000 7f621c7f0700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:08:29.790 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2021-02-11T10:08:29.791+0000 7f621a7ec700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:08:29.790 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2021-02-11T10:08:29.791+0000 7f62197ea700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:08:29.790 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2021-02-11T10:08:29.791+0000 7f62417fa700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:08:29.790 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2021-02-11T10:08:29.791+0000 7f8555ff3700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:08:29.791 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2021-02-11T10:08:29.791+0000 7f8551feb700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:08:29.791 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2021-02-11T10:08:29.791+0000 7f8567fff700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:08:29.791 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2021-02-11T10:08:29.791+0000 7f85557f2700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:08:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:30.021 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:08:30.065 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038109.580586_493.1","last_update":"2021-02-11T10:08:29.580586Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:08:30.065 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:08:30.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:30.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:30.066 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:30.102 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:30.102 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:30.356 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038109.580586_493.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:29.580586Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:08:30.356 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:08:30.356 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:08:30.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:30.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:30.357 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:30.398 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:30.398 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:30.659 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038109.580586_493.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:29.580586Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:08:30.659 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:08:30.659 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:08:30.659 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:30.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:30.660 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:30.699 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:30.699 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:30.944 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038109.580586_493.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:29.580586Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:08:30.945 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:08:30.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:30.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:30.945 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:31.034 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:31.034 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:31.287 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:31.287+0000 7f9ac49a6080 1 bucket sync caught up with source: 2021-02-11T10:08:31.287 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000001.33.5, , , , 00000000001.71.5, , , 00000000001.37.5, , 00000000001.52.5] 2021-02-11T10:08:31.287 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000001.33.5, , , , 00000000001.71.5, , , 00000000001.37.5, , 00000000001.52.5] 2021-02-11T10:08:31.287 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:31.287+0000 7f9ac49a6080 0 bucket checkpoint complete 2021-02-11T10:08:31.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:31.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:31.324 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:31.372 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:31.373 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:31.606 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:31.599+0000 7f2f24b0a080 1 bucket sync caught up with source: 2021-02-11T10:08:31.606 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.33.5, , , , 00000000001.71.5, , , 00000000001.37.5, , 00000000001.52.5] 2021-02-11T10:08:31.606 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.33.5, , , , 00000000001.71.5, , , 00000000001.37.5, , 00000000001.52.5] 2021-02-11T10:08:31.606 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:31.599+0000 7f2f24b0a080 0 bucket checkpoint complete 2021-02-11T10:08:31.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:31.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:31.639 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:31.687 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:31.687 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:31.924 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:31.923+0000 7f00b773d080 1 bucket sync caught up with source: 2021-02-11T10:08:31.924 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000001.33.5, , , , 00000000001.71.5, , , 00000000001.37.5, , 00000000001.52.5] 2021-02-11T10:08:31.924 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000001.33.5, , , , 00000000001.71.5, , , 00000000001.37.5, , 00000000001.52.5] 2021-02-11T10:08:31.924 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:31.923+0000 7f00b773d080 0 bucket checkpoint complete 2021-02-11T10:08:31.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:31.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:31.958 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:32.005 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:32.006 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:32.272 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:32.267+0000 7fc2559e9080 1 bucket sync caught up with source: 2021-02-11T10:08:32.273 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.39.5, , , , 00000000001.72.5, , , 00000000001.42.5, , 00000000001.55.5] 2021-02-11T10:08:32.273 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.39.5, , , , 00000000001.72.5, , , 00000000001.42.5, , 00000000001.55.5] 2021-02-11T10:08:32.273 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:32.267+0000 7fc2559e9080 0 bucket checkpoint complete 2021-02-11T10:08:32.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:32.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:32.309 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:32.356 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:32.357 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:32.604 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:32.595+0000 7f9e49ca5080 1 bucket sync caught up with source: 2021-02-11T10:08:32.604 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.39.5, , , , 00000000001.72.5, , , 00000000001.42.5, , 00000000001.55.5] 2021-02-11T10:08:32.604 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.39.5, , , , 00000000001.72.5, , , 00000000001.42.5, , 00000000001.55.5] 2021-02-11T10:08:32.604 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:32.595+0000 7f9e49ca5080 0 bucket checkpoint complete 2021-02-11T10:08:32.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:32.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:32.639 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-69 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:32.687 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:32.687 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:32.927 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:32.919+0000 7f7725ec8080 1 bucket sync caught up with source: 2021-02-11T10:08:32.927 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.36.5, , , , 00000000001.71.5, , , 00000000001.43.5, , 00000000001.55.5] 2021-02-11T10:08:32.927 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.71.5, , , 00000000001.43.5, , 00000000001.55.5] 2021-02-11T10:08:32.927 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:32.919+0000 7f7725ec8080 0 bucket checkpoint complete 2021-02-11T10:08:32.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:32.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:32.960 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-69 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:33.007 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:33.008 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:33.248 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:33.246+0000 7f5b6752e080 1 bucket sync caught up with source: 2021-02-11T10:08:33.248 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000001.36.5, , , , 00000000001.71.5, , , 00000000001.43.5, , 00000000001.55.5] 2021-02-11T10:08:33.248 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.71.5, , , 00000000001.43.5, , 00000000001.55.5] 2021-02-11T10:08:33.248 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:33.246+0000 7f5b6752e080 0 bucket checkpoint complete 2021-02-11T10:08:33.285 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-69 zones={test-zone2, test-zone1} 2021-02-11T10:08:34.906 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:08:34.914 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:08:34.914 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:08:34.914 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:08:34.914 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:08:34.914 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:08:34.921 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:08:34.921 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:08:34.921 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:08:34.921 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:08:34.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:08:34.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:08:35.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:08:35.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:08:35.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:08:35.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:08:35.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:08:35.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:08:35.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:08:35.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:08:35.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:08:35.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:08:36.022 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-69 zones={test-zone2, test-zone1} 2021-02-11T10:08:36.022 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-69 zones={test-zone3, test-zone1} 2021-02-11T10:08:36.375 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:08:36.381 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:08:36.381 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:08:36.381 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:08:36.381 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:08:36.381 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:08:36.385 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:08:36.385 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:08:36.385 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:08:36.385 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:08:36.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:08:36.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:08:36.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:08:36.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:08:36.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:08:36.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:08:36.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:08:36.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:08:37.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:08:37.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:08:37.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:08:37.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:08:37.474 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-69 zones={test-zone3, test-zone1} 2021-02-11T10:08:37.475 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-69 zones={test-zone3, test-zone2} 2021-02-11T10:08:37.531 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:08:37.541 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:08:37.541 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:08:37.541 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:08:37.541 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:08:37.541 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:08:37.547 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:08:37.547 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:08:37.547 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:08:37.547 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:08:37.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:08:37.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:08:37.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:08:37.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:08:37.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:08:37.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:08:38.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:08:38.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:08:38.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:08:38.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:08:38.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:08:38.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:08:38.626 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-69 zones={test-zone3, test-zone2} 2021-02-11T10:08:43.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:08:43.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:08:43.893 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:08:43.930 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:43.930 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:44.215 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-70 2021-02-11T10:08:44.596 INFO:tasks.rgw.c1.client.1.smithi026.stdout:2021-02-11T10:08:44.591+0000 7fd64dffb700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:08:44.597 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2021-02-11T10:08:44.594+0000 7f621e7f4700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:08:44.597 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2021-02-11T10:08:44.594+0000 7f85677fe700 -1 data sync zone:a3c4910a ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:08:44.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:44.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:44.751 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:44.794 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:44.795 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:45.023 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:08:45.066 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038124.551195_504.1","last_update":"2021-02-11T10:08:44.551195Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:08:45.066 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:08:45.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:45.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:45.067 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:45.099 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:45.099 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:45.330 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038124.551195_504.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.551195Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:08:45.330 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:08:45.330 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:08:45.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:45.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:45.331 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:45.371 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:45.371 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:45.635 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038124.551195_504.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.551195Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:08:45.636 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:08:45.636 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:08:45.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:45.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:45.636 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:45.676 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:45.676 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:45.925 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038124.551195_504.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.551195Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:08:45.926 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:08:45.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:45.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:45.926 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:46.015 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:46.016 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:46.254 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:46.254+0000 7fe25a89f080 1 bucket sync caught up with source: 2021-02-11T10:08:46.254 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000001.75.5, , , , 00000000001.55.5, , , 00000000001.61.5, , 00000000001.66.5] 2021-02-11T10:08:46.255 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000001.75.5, , , , 00000000001.55.5, , , 00000000001.61.5, , 00000000001.66.5] 2021-02-11T10:08:46.255 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:46.254+0000 7fe25a89f080 0 bucket checkpoint complete 2021-02-11T10:08:46.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:46.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:46.303 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:46.353 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:46.353 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:46.606 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:46.599+0000 7f66bdc39080 1 bucket sync caught up with source: 2021-02-11T10:08:46.606 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.75.5, , , , 00000000001.55.5, , , 00000000001.61.5, , 00000000001.66.5] 2021-02-11T10:08:46.606 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.75.5, , , , 00000000001.55.5, , , 00000000001.61.5, , 00000000001.66.5] 2021-02-11T10:08:46.606 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:46.599+0000 7f66bdc39080 0 bucket checkpoint complete 2021-02-11T10:08:46.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:46.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:46.637 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:46.679 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:46.679 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:46.913 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:46.914+0000 7fcf77a08080 1 waiting for incremental sync to catch up: 2021-02-11T10:08:46.914 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000001.75.5, , , , , , , 00000000001.61.5, , 00000000001.66.5] 2021-02-11T10:08:46.914 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000001.75.5, , , , 00000000001.55.5, , , 00000000001.61.5, , 00000000001.66.5] 2021-02-11T10:08:51.919 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:51.918+0000 7fcf77a08080 1 bucket sync caught up with source: 2021-02-11T10:08:51.919 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000001.75.5, , , , 00000000001.55.5, , , 00000000001.61.5, , 00000000001.66.5] 2021-02-11T10:08:51.919 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000001.75.5, , , , 00000000001.55.5, , , 00000000001.61.5, , 00000000001.66.5] 2021-02-11T10:08:51.919 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:51.918+0000 7fcf77a08080 0 bucket checkpoint complete 2021-02-11T10:08:51.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:51.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:51.955 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:52.002 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:52.003 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:52.257 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:52.251+0000 7fc7353be080 1 bucket sync caught up with source: 2021-02-11T10:08:52.257 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.78.5, , , , 00000000001.59.5, , , 00000000001.64.5, , 00000000001.75.5] 2021-02-11T10:08:52.257 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.78.5, , , , 00000000001.59.5, , , 00000000001.64.5, , 00000000001.75.5] 2021-02-11T10:08:52.257 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:52.251+0000 7fc7353be080 0 bucket checkpoint complete 2021-02-11T10:08:52.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:52.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:52.295 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:52.341 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:52.341 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:52.583 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:52.575+0000 7ff6f75ce080 1 bucket sync caught up with source: 2021-02-11T10:08:52.583 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.78.5, , , , 00000000001.59.5, , , 00000000001.64.5, , 00000000001.75.5] 2021-02-11T10:08:52.583 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.78.5, , , , 00000000001.59.5, , , 00000000001.64.5, , 00000000001.75.5] 2021-02-11T10:08:52.583 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:52.575+0000 7ff6f75ce080 0 bucket checkpoint complete 2021-02-11T10:08:52.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:52.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:52.620 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-70 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:52.665 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:52.665 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:52.899+0000 7fa602455080 1 bucket sync caught up with source: 2021-02-11T10:08:52.906 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, 00000000001.77.5, , , , 00000000001.59.5, , , 00000000001.63.5, , 00000000001.65.5] 2021-02-11T10:08:52.906 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, 00000000001.77.5, , , , 00000000001.59.5, , , 00000000001.63.5, , 00000000001.65.5] 2021-02-11T10:08:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:08:52.899+0000 7fa602455080 0 bucket checkpoint complete 2021-02-11T10:08:52.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:52.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:08:52.935 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-70 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:08:52.981 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:52.981 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:53.229 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:53.230+0000 7fd46f716080 1 bucket sync caught up with source: 2021-02-11T10:08:53.229 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000001.77.5, , , , 00000000001.59.5, , , 00000000001.63.5, , 00000000001.65.5] 2021-02-11T10:08:53.229 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000001.77.5, , , , 00000000001.59.5, , , 00000000001.63.5, , 00000000001.65.5] 2021-02-11T10:08:53.229 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:08:53.230+0000 7fd46f716080 0 bucket checkpoint complete 2021-02-11T10:08:53.264 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-70 zones={test-zone2, test-zone1} 2021-02-11T10:08:53.956 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:08:53.964 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:08:53.964 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:08:53.964 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:08:53.964 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:08:53.964 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:08:53.971 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:08:53.971 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:08:53.971 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:08:53.971 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:08:53.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:08:54.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:08:54.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:08:54.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:08:54.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:08:54.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:08:54.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:08:54.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:08:54.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:08:54.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:08:54.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:08:54.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:08:55.058 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-70 zones={test-zone2, test-zone1} 2021-02-11T10:08:55.058 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-70 zones={test-zone3, test-zone1} 2021-02-11T10:08:55.649 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:08:55.654 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:08:55.654 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:08:55.654 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:08:55.654 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:08:55.655 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:08:55.659 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:08:55.659 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:08:55.659 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:08:55.659 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:08:55.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:08:55.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:08:55.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:08:55.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:08:55.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:08:56.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:08:56.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:08:56.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:08:56.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:08:56.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:08:56.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:08:56.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:08:56.752 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-70 zones={test-zone3, test-zone1} 2021-02-11T10:08:56.753 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-70 zones={test-zone3, test-zone2} 2021-02-11T10:08:56.821 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:08:56.828 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:08:56.828 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:08:56.828 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:08:56.828 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:08:56.828 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:08:56.835 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:08:56.835 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:08:56.835 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:08:56.835 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:08:56.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:08:56.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:08:57.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:08:57.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:08:57.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:08:57.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:08:57.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:08:57.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:08:57.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:08:57.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:08:57.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:08:57.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:08:57.937 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=erivkc-70 zones={test-zone3, test-zone2} 2021-02-11T10:08:57.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:08:57.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:08:57.937 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:08:57.977 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:57.977 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:58.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'erivkc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:58.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'erivkc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:58.281 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket erivkc-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:58.363 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:58.363 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:58.632 INFO:teuthology.orchestra.run.smithi026.stdout:[] 2021-02-11T10:08:58.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'erivkc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:58.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'erivkc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:58.633 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket erivkc-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:58.713 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:58.713 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:58.968 INFO:teuthology.orchestra.run.smithi026.stdout:[{"op_id":"1#00000000001.33.5","op_tag":"a3c4910a-4529-4294-aed4-ef409f209cf5.5475.4674","op":"write","object":"a","instance":"","state":"complete","index_ver":1,"timestamp":"2021-02-11T10:08:29.592634205Z","ver":{"pool":9,"epoch":1},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-69:a3c4910a-4529-4294-aed4-ef409f209cf5.5479.20"}]},{"op_id":"5#00000000001.71.5","op_tag":"a3c4910a-4529-4294-aed4-ef409f209cf5.5475.4685","op":"write","object":"d","instance":"","state":"complete","index_ver":1,"timestamp":"2021-02-11T10:08:29.736960483Z","ver":{"pool":9,"epoch":5},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-69:a3c4910a-4529-4294-aed4-ef409f209cf5.5479.20"}]},{"op_id":"8#00000000001.37.5","op_tag":"a3c4910a-4529-4294-aed4-ef409f209cf5.5475.4681","op":"write","object":"c","instance":"","state":"complete","index_ver":1,"timestamp":"2021-02-11T10:08:29.687868465Z","ver":{"pool":9,"epoch":4},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-69:a3c4910a-4529-4294-aed4-ef409f209cf5.5479.20"}]},{"op_id":"10#00000000001.52.5","op_tag":"a3c4910a-4529-4294-aed4-ef409f209cf5.5475.4678","op":"write","object":"b","instance":"","state":"complete","index_ver":1,"timestamp":"2021-02-11T10:08:29.640765434Z","ver":{"pool":9,"epoch":3},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"a3c4910a-4529-4294-aed4-ef409f209cf5:erivkc-69:a3c4910a-4529-4294-aed4-ef409f209cf5.5479.20"}]}] 2021-02-11T10:08:58.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:08:58.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:08:58.969 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:08:59.050 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:59.050 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:00.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'erivkc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:00.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'erivkc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:00.608 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket erivkc-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:00.689 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:00.689 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:00.967 INFO:teuthology.orchestra.run.smithi026.stdout:[] 2021-02-11T10:09:00.968 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... ok 2021-02-11T10:09:02.801 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_creation_time ... ok 2021-02-11T10:09:02.801 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=erivkc-71 2021-02-11T10:09:02.863 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=erivkc-72 2021-02-11T10:09:02.945 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=erivkc-73 2021-02-11T10:09:03.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:03.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:03.187 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:03.230 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:03.231 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:03.425 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:09:03.465 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038143.004002_524.1","last_update":"2021-02-11T10:09:03.004002Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:09:03.466 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:09:03.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:03.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:03.466 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:03.504 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:03.504 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:03.717 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038143.004002_524.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.004002Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:09:03.717 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:09:03.717 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:09:03.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:03.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:03.718 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:03.752 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:03.752 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:04.009 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038143.004002_524.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.004002Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:09:04.010 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:09:04.010 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:09:04.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:04.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:04.010 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:04.050 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:04.050 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:04.294 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038143.004002_524.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.004002Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:09:04.295 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:09:04.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:04.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:04.295 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:04.341 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:04.341 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:04.616 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613038124.670147_1001.1","last_update":"2021-02-11T10:08:44.670147Z"},{"marker":"1_1613038124.717685_640.1","last_update":"2021-02-11T10:08:44.717685Z"},{"marker":"1_1613038124.622684_339.1","last_update":"2021-02-11T10:08:44.622684Z"},{"marker":"1_1613038143.374527_523.1","last_update":"2021-02-11T10:09:03.374527Z"}] 2021-02-11T10:09:04.616 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:09:04.617 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:04.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:04.617 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:04.659 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:04.659 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:04.916 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613038143.374527_523.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.374527Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:04.917 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:09:04.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:04.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:04.917 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:04.958 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:04.958 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:05.228 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613038124.670147_1001.1","last_update":"2021-02-11T10:08:44.670147Z"},{"marker":"1_1613038124.717685_640.1","last_update":"2021-02-11T10:08:44.717685Z"},{"marker":"1_1613038124.622684_339.1","last_update":"2021-02-11T10:08:44.622684Z"},{"marker":"1_1613038143.374527_523.1","last_update":"2021-02-11T10:09:03.374527Z"}] 2021-02-11T10:09:05.228 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T10:09:05.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:05.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:05.229 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:05.315 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:05.315 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:05.579 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7045610624405686142},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613038143.374527_523.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.374527Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:05.579 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T10:09:05.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:05.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:05.580 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:05.658 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:05.658 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:05.923 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613038124.670147_1001.1","last_update":"2021-02-11T10:08:44.670147Z"},{"marker":"1_1613038124.717685_640.1","last_update":"2021-02-11T10:08:44.717685Z"},{"marker":"1_1613038124.622684_339.1","last_update":"2021-02-11T10:08:44.622684Z"},{"marker":"1_1613038143.374527_523.1","last_update":"2021-02-11T10:09:03.374527Z"}] 2021-02-11T10:09:05.923 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T10:09:05.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:05.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:05.924 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:05.965 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:05.965 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:06.226 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1531294025241915737},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613038143.374527_523.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.374527Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:06.226 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T10:09:06.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:06.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:06.226 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:06.306 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:06.306 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:06.570 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1613038124.917800_929.1","last_update":"2021-02-11T10:08:44.917800Z"},{"marker":"1_1613038124.964439_582.1","last_update":"2021-02-11T10:08:44.964439Z"},{"marker":"1_1613038124.920171_316.1","last_update":"2021-02-11T10:08:44.920171Z"},{"marker":"1_1613038143.373419_688.1","last_update":"2021-02-11T10:09:03.373419Z"}] 2021-02-11T10:09:06.570 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T10:09:06.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:06.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:06.570 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:06.615 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:06.615 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:06.896 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3492294689851219663},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.917800_929.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.917800Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.964439_582.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.964439Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.920171_316.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.920171Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613038143.373419_688.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.373419Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:06.896 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T10:09:06.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:06.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:06.897 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:06.936 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:06.936 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:07.193 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1613038124.917800_929.1","last_update":"2021-02-11T10:08:44.917800Z"},{"marker":"1_1613038124.964439_582.1","last_update":"2021-02-11T10:08:44.964439Z"},{"marker":"1_1613038124.920171_316.1","last_update":"2021-02-11T10:08:44.920171Z"},{"marker":"1_1613038143.373419_688.1","last_update":"2021-02-11T10:09:03.373419Z"}] 2021-02-11T10:09:07.193 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T10:09:07.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:07.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:07.194 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:07.236 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:07.236 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:07.517 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7442380677805818242},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.917800_929.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.917800Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.964439_582.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.964439Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.920171_316.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.920171Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613038143.373419_688.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.373419Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:07.517 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T10:09:07.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:07.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:07.518 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:07.600 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:07.600 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:07.883 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613038124.875374_899.1","last_update":"2021-02-11T10:08:44.875374Z"},{"marker":"1_1613038124.920409_391.1","last_update":"2021-02-11T10:08:44.920409Z"},{"marker":"1_1613038125.001960_221.1","last_update":"2021-02-11T10:08:45.001960Z"},{"marker":"1_1613038143.369145_470.1","last_update":"2021-02-11T10:09:03.369145Z"}] 2021-02-11T10:09:07.884 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T10:09:07.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:07.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:07.884 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:07.967 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:07.967 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:08.237 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14314976040552073418},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.875374_899.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.875374Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.920409_391.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.920409Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038125.001960_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:45.001960Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613038143.369145_470.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.369145Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:08.237 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T10:09:08.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:08.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:08.237 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:08.321 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:08.321 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:08.583 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"1_1613038124.875374_899.1","last_update":"2021-02-11T10:08:44.875374Z"},{"marker":"1_1613038124.920409_391.1","last_update":"2021-02-11T10:08:44.920409Z"},{"marker":"1_1613038125.001960_221.1","last_update":"2021-02-11T10:08:45.001960Z"},{"marker":"1_1613038143.369145_470.1","last_update":"2021-02-11T10:09:03.369145Z"}] 2021-02-11T10:09:08.583 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T10:09:08.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:08.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:08.584 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:08.625 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:08.625 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:08.905 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12589397483957797345},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.875374_899.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.875374Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.920409_391.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.920409Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038125.001960_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:45.001960Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613038143.369145_470.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.369145Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:08.905 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T10:09:08.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'type', '--log-type', 'fifo', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:09:08.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'type', '--log-type', 'fifo', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:09:08.906 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog type --log-type fifo --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:09:08.944 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:08.945 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:10.239 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'type', '--log-type', 'fifo', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:09:21.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'type', '--log-type', 'fifo', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:09:21.069 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog type --log-type fifo --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:09:21.108 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:21.109 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:22.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'type', '--log-type', 'fifo', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:09:22.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'type', '--log-type', 'fifo', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:09:22.347 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog type --log-type fifo --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:09:22.387 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:22.388 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:23.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:23.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:23.479 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:23.514 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:23.514 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:23.712 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:09:23.763 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038143.004002_524.1","last_update":"2021-02-11T10:09:03.004002Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:09:23.764 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:09:23.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:23.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:23.764 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:23.805 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:23.805 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:24.079 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038143.004002_524.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.004002Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:09:24.080 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:09:24.080 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:09:24.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:24.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:24.080 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:24.123 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:24.123 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:24.390 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038143.004002_524.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.004002Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:09:24.391 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:09:24.391 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:09:24.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:24.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:24.391 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:24.430 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:24.430 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:24.664 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038143.004002_524.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.004002Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:09:24.665 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:09:24.665 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:24.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:24.665 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:24.993 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"G00000000000000000001@00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"G00000000000000000001@00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"G00000000000000000001@00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"G00000000000000000001@00000000000000000000:00000000000000000000","last_update":"0.000000"}] 2021-02-11T10:09:24.993 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:09:24.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:24.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:24.994 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:25.035 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:25.036 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:25.281 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613038143.374527_523.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.374527Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:25.281 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038143.374527_523.1 2021-02-11T10:09:30.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:30.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:30.283 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:30.324 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:30.324 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:30.580 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613038143.374527_523.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.374527Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:30.581 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038143.374527_523.1 2021-02-11T10:09:35.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:35.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:35.583 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:35.634 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:35.635 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:35.889 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613038143.374527_523.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.374527Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:35.890 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038143.374527_523.1 2021-02-11T10:09:40.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:42.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:42.352 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:42.395 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:42.396 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:42.641 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:42.642 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:09:47.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:47.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:47.643 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:47.685 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:47.685 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:47.937 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:47.938 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:09:52.939 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:52.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:52.940 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:52.987 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:52.988 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:53.258 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:53.258 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:09:58.259 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:58.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:58.260 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:58.303 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:58.303 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:58.603 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:58.604 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:10:03.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:03.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:03.606 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:10:03.644 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:10:03.644 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:10:03.899 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:10:03.899 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:10:08.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:08.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:08.900 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:10:08.943 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:10:08.944 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:10:09.206 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:10:09.206 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:10:14.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:14.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:14.208 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:10:14.250 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:10:14.250 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:10:14.524 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:10:14.525 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:10:19.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:19.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:19.526 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:10:19.572 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:10:19.573 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:10:19.825 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:10:19.825 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:10:24.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:24.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:24.827 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:10:24.871 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:10:24.871 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:10:25.149 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:10:25.149 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:10:30.150 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:30.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:30.151 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:10:30.192 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:10:30.193 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:10:30.455 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:10:30.455 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:10:35.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:35.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:35.457 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:10:35.506 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:10:35.506 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:10:35.757 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:10:35.758 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:10:40.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:40.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:40.759 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:10:40.803 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:10:40.803 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:10:41.087 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:10:41.087 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:10:46.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:46.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:46.089 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:10:46.132 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:10:46.132 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:10:46.379 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:10:46.379 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:10:51.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:51.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:51.381 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:10:51.423 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:10:51.424 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:10:51.670 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:10:51.671 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:10:56.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:56.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:56.672 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:10:56.715 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:10:56.715 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:10:56.991 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:10:56.992 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:11:01.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:01.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:01.993 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:02.036 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:02.036 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:02.306 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:02.307 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:11:07.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:07.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:07.308 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:07.351 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:07.351 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:07.605 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:07.605 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:11:12.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:12.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:12.606 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:12.645 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:12.646 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:12.914 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:12.914 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:11:17.915 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:17.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:17.916 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:17.960 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:17.960 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:18.207 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:18.207 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:11:23.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:23.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:23.208 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:23.249 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:23.250 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:23.509 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:23.509 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:11:28.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:28.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:28.510 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:28.553 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:28.553 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:28.829 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:28.830 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:11:33.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:33.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:33.831 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:33.873 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:33.873 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:34.148 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:34.148 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:11:39.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:39.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:39.150 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:39.193 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:39.193 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:39.453 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:39.453 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:11:44.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:44.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:44.455 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:44.500 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:44.501 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:44.760 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:44.761 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:11:49.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:49.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:49.763 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:49.806 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:49.806 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:50.073 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:50.073 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:11:55.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:55.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:55.075 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:55.120 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:55.120 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:55.401 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:55.401 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:12:00.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:00.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:00.485 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:12:00.527 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:12:00.528 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:12:00.812 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:12:00.813 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:12:05.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:05.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:05.814 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:12:05.856 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:12:05.856 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:12:06.109 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:12:06.110 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:12:11.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:11.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:11.112 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:12:11.154 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:12:11.154 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:12:11.425 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:12:11.426 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:12:16.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:16.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:16.427 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:12:16.469 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:12:16.469 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:12:16.741 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:12:16.742 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:12:21.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:21.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:21.743 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:12:21.784 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:12:21.785 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:12:22.049 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:12:22.050 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:12:27.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:27.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:27.052 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:12:27.094 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:12:27.094 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:12:27.366 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:12:27.366 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:12:32.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:32.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:32.367 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:12:32.410 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:12:32.410 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:12:32.678 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:12:32.679 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:12:37.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:37.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:37.680 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:12:37.723 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:12:37.723 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:12:37.975 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:12:37.975 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:12:42.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:42.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:42.977 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:12:43.020 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:12:43.020 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:12:43.283 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:12:43.283 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:12:48.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:48.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:48.285 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:12:48.327 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:12:48.327 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:12:48.589 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:12:48.589 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:12:53.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:53.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:53.590 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:12:53.632 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:12:53.632 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:12:53.889 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:12:53.890 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:12:58.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:58.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:58.892 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:12:58.934 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:12:58.935 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:12:59.206 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:12:59.207 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:13:04.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:04.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:04.208 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:04.253 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:04.253 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:04.525 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:13:04.525 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:13:09.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:09.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:09.526 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:09.568 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:09.568 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:09.833 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:13:09.833 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:13:14.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:14.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:14.835 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:14.876 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:14.876 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:15.131 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:13:15.131 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:13:20.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:20.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:20.133 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:20.176 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:20.176 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:20.449 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:13:20.450 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:13:25.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:25.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:25.451 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:25.494 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:25.494 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:25.756 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:13:25.756 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:13:30.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:30.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:30.757 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:30.801 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:30.802 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:31.068 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:13:31.068 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:13:36.069 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:36.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:36.070 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:36.113 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:36.113 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:36.390 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:13:36.390 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:13:41.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:41.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:41.392 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:41.435 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:41.435 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:41.723 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:13:41.724 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:13:46.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:46.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:46.724 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:46.767 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:46.767 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:47.032 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:13:47.033 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:13:52.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:52.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:52.034 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:52.078 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:52.078 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:52.353 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:13:52.353 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:13:57.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:57.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:57.354 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:57.398 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:57.398 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:57.676 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:13:57.676 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:14:02.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:02.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:02.677 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:02.721 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:02.721 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:02.994 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:14:02.994 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:14:07.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:07.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:07.995 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:08.038 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:08.039 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:08.296 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:14:08.296 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:14:13.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:13.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:13.298 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:13.341 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:13.341 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:13.613 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:14:13.614 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:14:18.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:18.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:18.615 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:18.655 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:18.655 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:18.928 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:14:18.929 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:14:23.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:23.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:23.931 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:23.974 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:23.975 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:24.243 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:14:24.243 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:14:29.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:29.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:29.245 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:29.289 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:29.289 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:29.566 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:14:29.566 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:14:34.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:34.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:34.568 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:34.607 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:34.607 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:34.864 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:14:34.864 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:14:39.865 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:39.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:39.866 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:39.910 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:39.910 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:40.173 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2845144745394880255},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613038124.670147_1001.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.670147Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613038124.717685_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.717685Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613038124.622684_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:08:44.622684Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:28.279452Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:14:40.174 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.670147_1001.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.717685_640.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613038124.622684_339.1 2021-02-11T10:14:45.176 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_backing ... FAIL 2021-02-11T10:14:45.177 INFO:tasks.rgw_multisite_tests: 2021-02-11T10:14:45.177 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T10:14:45.177 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_datalog_backing 2021-02-11T10:14:45.177 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T10:14:45.177 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T10:14:45.177 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T10:14:45.177 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T10:14:45.177 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa/tasks/rgw_multi/tests.py", line 1371, in test_datalog_backing 2021-02-11T10:14:45.177 INFO:tasks.rgw_multisite_tests: zonegroup_data_checkpoint(zonegroup_conns) 2021-02-11T10:14:45.178 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa/tasks/rgw_multi/tests.py", line 367, in zonegroup_data_checkpoint 2021-02-11T10:14:45.178 INFO:tasks.rgw_multisite_tests: zone_data_checkpoint(target_conn.zone, source_conn.zone) 2021-02-11T10:14:45.178 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa/tasks/rgw_multi/tests.py", line 359, in zone_data_checkpoint 2021-02-11T10:14:45.178 INFO:tasks.rgw_multisite_tests: (target_zone.name, source_zone.name) 2021-02-11T10:14:45.178 INFO:tasks.rgw_multisite_tests:AssertionError: failed data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:14:45.178 INFO:tasks.rgw_multisite_tests: 2021-02-11T10:14:45.178 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T10:14:45.178 INFO:tasks.rgw_multisite_tests:Ran 29 tests in 1523.672s 2021-02-11T10:14:45.178 INFO:tasks.rgw_multisite_tests: 2021-02-11T10:14:45.178 INFO:tasks.rgw_multisite_tests:FAILED (failures=1) 2021-02-11T10:14:45.182 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2021-02-11T10:14:45.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:45.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:45.183 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:45.222 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:45.222 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:45.449 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:14:45.491 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038143.004002_524.1","last_update":"2021-02-11T10:09:03.004002Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:45.491 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:14:45.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:45.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:45.491 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:45.535 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:45.535 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:45.805 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038143.004002_524.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.004002Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:45.806 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:14:45.806 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:14:45.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:45.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:45.806 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:45.847 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:45.847 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:46.121 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038143.004002_524.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.004002Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:46.122 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:14:46.122 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:46.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:46.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:46.122 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:46.163 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:46.163 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:46.418 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038143.004002_524.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.004002Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:46.419 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:46.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:46.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:46.419 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:46.462 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:46.462 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:46.687 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:14:46.729 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038143.004002_524.1","last_update":"2021-02-11T10:09:03.004002Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:46.729 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:46.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:46.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:46.730 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:46.770 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:46.770 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:47.029 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038143.004002_524.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:09:03.004002Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:47.029 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:47.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:47.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:47.085 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:47.123 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:47.123 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:47.345 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:14:47.389 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038487.079288_548.1","last_update":"2021-02-11T10:14:47.079288Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:47.389 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:47.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:47.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:47.390 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:47.430 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:47.430 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:47.673 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038487.079288_548.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:47.079288Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:47.673 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:47.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:47.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:47.781 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:47.822 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:47.822 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:48.050 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:14:48.092 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038487.079288_548.1","last_update":"2021-02-11T10:14:47.079288Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:48.092 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:48.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:48.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:48.092 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:48.131 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:48.131 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:48.368 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038487.079288_548.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:47.079288Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:48.369 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:49.340 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... ok 2021-02-11T10:14:49.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:49.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:49.341 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:49.380 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:49.380 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:49.600 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:14:49.643 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038489.325706_552.1","last_update":"2021-02-11T10:14:49.325706Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:49.643 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:14:49.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:49.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:49.644 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:49.687 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:49.687 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:49.967 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038489.325706_552.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:49.325706Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:49.968 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:14:49.968 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:14:49.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:49.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:49.968 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:50.013 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:50.013 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:50.276 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038489.325706_552.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:49.325706Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:50.276 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:14:50.276 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:50.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:50.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:50.277 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:50.318 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:50.318 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:50.559 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038489.325706_552.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:49.325706Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:50.560 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:50.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:50.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:50.560 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:50.600 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:50.600 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:50.818 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:14:50.859 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038489.325706_552.1","last_update":"2021-02-11T10:14:49.325706Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:50.859 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:50.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:50.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:50.859 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:50.899 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:50.899 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:51.152 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038489.325706_552.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:49.325706Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:51.152 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:51.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:51.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:51.283 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:51.320 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:51.321 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:51.533 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:14:51.576 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038491.277588_558.1","last_update":"2021-02-11T10:14:51.277588Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:51.577 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:51.577 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:51.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:51.577 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:51.620 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:51.620 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:51.866 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038491.277588_558.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:51.277588Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:51.867 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:51.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:51.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:51.907 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:51.947 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:51.947 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:52.178 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:14:52.224 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038491.277588_558.1","last_update":"2021-02-11T10:14:51.277588Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:52.224 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:52.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:52.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:52.225 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:52.267 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:52.268 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:52.504 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038491.277588_558.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:51.277588Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:52.505 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:53.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:53.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:53.127 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:14:53.189 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:53.189 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:53.409 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:14:53.407+0000 7ffbc88d1080 1 bucket sync caught up with source: 2021-02-11T10:14:53.409 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000002.115.5, , 00000000001.102.5, 00000000001.111.5, 00000000001.58.5, , 00000000001.80.5, 00000000001.86.5, 00000000001.104.5, 00000000002.78.5] 2021-02-11T10:14:53.410 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000002.115.5, , 00000000001.102.5, 00000000001.111.5, 00000000001.58.5, , 00000000001.80.5, 00000000001.86.5, 00000000001.104.5, 00000000002.78.5] 2021-02-11T10:14:53.410 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:14:53.407+0000 7ffbc88d1080 0 bucket checkpoint complete 2021-02-11T10:14:53.447 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:14:52.713287Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'erivkc-75_notif', 'bucket': {'name': 'erivkc-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::erivkc-75', 'id': 'a3c4910a-4529-4294-aed4-ef409f209cf5.5479.26'}, 'object': {'key': '0', 'size': 0, 'etag': '71cdf987', 'versionId': '', 'sequencer': '9C0325601D809234', 'metadata': [], 'tags': []}}, 'eventId': '1613038492.882016.71cdf987', 'opaqueData': ''} 2021-02-11T10:14:53.448 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:14:52.776548Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'erivkc-75_notif', 'bucket': {'name': 'erivkc-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::erivkc-75', 'id': 'a3c4910a-4529-4294-aed4-ef409f209cf5.5479.26'}, 'object': {'key': '2', 'size': 0, 'etag': '9d66f71d', 'versionId': '', 'sequencer': '9C03256089620437', 'metadata': [], 'tags': []}}, 'eventId': '1613038492.923034.9d66f71d', 'opaqueData': ''} 2021-02-11T10:14:53.448 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:14:52.732314Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'erivkc-75_notif', 'bucket': {'name': 'erivkc-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::erivkc-75', 'id': 'a3c4910a-4529-4294-aed4-ef409f209cf5.5479.26'}, 'object': {'key': '1', 'size': 0, 'etag': '5cad0d0d', 'versionId': '', 'sequencer': '9C03256038CDB239', 'metadata': [], 'tags': []}}, 'eventId': '1613038492.968019.5cad0d0d', 'opaqueData': ''} 2021-02-11T10:14:53.448 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:14:52.996505Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'erivkc-75_notif', 'bucket': {'name': 'erivkc-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::erivkc-75', 'id': 'a3c4910a-4529-4294-aed4-ef409f209cf5.5479.26'}, 'object': {'key': '7', 'size': 0, 'etag': '14fbe30d', 'versionId': '', 'sequencer': '9D032560D413CC04', 'metadata': [], 'tags': []}}, 'eventId': '1613038493.080483.14fbe30d', 'opaqueData': ''} 2021-02-11T10:14:53.448 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:14:52.952295Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'erivkc-75_notif', 'bucket': {'name': 'erivkc-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::erivkc-75', 'id': 'a3c4910a-4529-4294-aed4-ef409f209cf5.5479.26'}, 'object': {'key': '6', 'size': 0, 'etag': 'bf4cb5ce', 'versionId': '', 'sequencer': '9D03256052B6CE04', 'metadata': [], 'tags': []}}, 'eventId': '1613038493.080655.bf4cb5ce', 'opaqueData': ''} 2021-02-11T10:14:53.448 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:14:52.908517Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'erivkc-75_notif', 'bucket': {'name': 'erivkc-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::erivkc-75', 'id': 'a3c4910a-4529-4294-aed4-ef409f209cf5.5479.26'}, 'object': {'key': '5', 'size': 0, 'etag': '87555c5f', 'versionId': '', 'sequencer': '9D032560C1CDA607', 'metadata': [], 'tags': []}}, 'eventId': '1613038493.128372.87555c5f', 'opaqueData': ''} 2021-02-11T10:14:53.448 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:14:53.084569Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'erivkc-75_notif', 'bucket': {'name': 'erivkc-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::erivkc-75', 'id': 'a3c4910a-4529-4294-aed4-ef409f209cf5.5479.26'}, 'object': {'key': '9', 'size': 0, 'etag': 'a5ae8727', 'versionId': '', 'sequencer': '9D032560A802170C', 'metadata': [], 'tags': []}}, 'eventId': '1613038493.202834.a5ae8727', 'opaqueData': ''} 2021-02-11T10:14:53.448 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:14:53.040139Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'erivkc-75_notif', 'bucket': {'name': 'erivkc-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::erivkc-75', 'id': 'a3c4910a-4529-4294-aed4-ef409f209cf5.5479.26'}, 'object': {'key': '8', 'size': 0, 'etag': 'c7dc07e8', 'versionId': '', 'sequencer': '9D03256094088F10', 'metadata': [], 'tags': []}}, 'eventId': '1613038493.277809.c7dc07e8', 'opaqueData': ''} 2021-02-11T10:14:53.448 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:14:52.864302Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'erivkc-75_notif', 'bucket': {'name': 'erivkc-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::erivkc-75', 'id': 'a3c4910a-4529-4294-aed4-ef409f209cf5.5479.26'}, 'object': {'key': '4', 'size': 0, 'etag': 'b873d6e5', 'versionId': '', 'sequencer': '9D03256005D79010', 'metadata': [], 'tags': []}}, 'eventId': '1613038493.277927.b873d6e5', 'opaqueData': ''} 2021-02-11T10:14:53.448 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:14:52.820583Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'erivkc-75_notif', 'bucket': {'name': 'erivkc-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::erivkc-75', 'id': 'a3c4910a-4529-4294-aed4-ef409f209cf5.5479.26'}, 'object': {'key': '3', 'size': 0, 'etag': 'bfbee389', 'versionId': '', 'sequencer': '9D032560B5189812', 'metadata': [], 'tags': []}}, 'eventId': '1613038493.311957.bfbee389', 'opaqueData': ''} 2021-02-11T10:14:53.458 WARNING:tasks.rgw_multi.tests_ps:superfluous records are found 2021-02-11T10:14:53.697 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... ok 2021-02-11T10:14:53.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:53.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:53.697 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:53.732 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:53.733 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:53.930 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:14:53.969 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038493.690537_566.1","last_update":"2021-02-11T10:14:53.690537Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:53.969 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:14:53.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:53.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:53.969 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:54.010 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:54.010 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:54.268 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038493.690537_566.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:53.690537Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:54.268 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:14:54.268 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:14:54.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:54.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:54.269 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:54.309 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:54.309 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:54.573 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038493.690537_566.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:53.690537Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:54.574 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:14:54.574 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:54.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:54.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:54.574 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:54.608 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:54.608 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:54.837 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038493.690537_566.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:53.690537Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:54.837 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:54.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:54.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:54.838 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:54.878 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:54.878 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:55.118 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:14:55.161 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038493.690537_566.1","last_update":"2021-02-11T10:14:53.690537Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:55.161 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:55.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:55.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:55.162 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:55.203 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:55.204 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:55.445 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038493.690537_566.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:53.690537Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:55.445 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:56.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:56.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:56.363 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:56.398 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:56.399 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:56.610 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:14:56.651 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038496.354322_573.1","last_update":"2021-02-11T10:14:56.354322Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:56.651 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:56.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:56.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:56.652 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:56.692 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:56.692 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:56.935 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038496.354322_573.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:56.354322Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:56.936 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:57.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:57.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:57.005 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:57.045 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:57.045 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:57.283 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:14:57.333 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038496.354322_573.1","last_update":"2021-02-11T10:14:56.354322Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:57.334 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:57.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:57.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:57.334 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:57.374 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:57.374 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:57.636 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038496.354322_573.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:56.354322Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:57.637 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:58.533 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... ok 2021-02-11T10:14:58.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:58.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:58.534 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:58.575 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:58.575 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:58.793 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:14:58.831 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038498.517536_577.1","last_update":"2021-02-11T10:14:58.517536Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:58.832 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:14:58.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:58.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:58.832 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:58.871 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:58.871 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:59.145 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038498.517536_577.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:58.517536Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:59.146 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:14:59.146 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:14:59.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:59.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:59.147 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:59.189 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:59.189 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:59.452 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038498.517536_577.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:58.517536Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:59.452 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:14:59.453 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:59.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:59.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:59.453 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:59.494 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:59.494 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:59.739 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038498.517536_577.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:58.517536Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:59.740 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:59.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:59.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:59.740 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:59.786 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:59.786 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:00.005 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:00.045 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038498.517536_577.1","last_update":"2021-02-11T10:14:58.517536Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:00.045 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:00.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:00.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:00.046 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:00.086 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:00.086 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:00.329 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038498.517536_577.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:58.517536Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:00.329 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:00.537 INFO:tasks.rgw_multisite_tests:test s3 topics set/get/delete on master ... ok 2021-02-11T10:15:00.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:00.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:00.538 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:00.579 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:00.579 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:00.819 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:00.862 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038498.517536_577.1","last_update":"2021-02-11T10:14:58.517536Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:00.862 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:15:00.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:00.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:00.863 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:00.904 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:00.904 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:01.173 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038498.517536_577.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:58.517536Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:01.173 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:15:01.174 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:15:01.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:01.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:01.174 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:01.214 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:01.214 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:01.484 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038498.517536_577.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:58.517536Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:01.484 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:15:01.484 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:01.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:01.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:01.485 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:01.525 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:01.525 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:01.768 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038498.517536_577.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:58.517536Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:01.768 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:01.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:01.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:01.769 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:01.804 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:01.804 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:02.021 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:02.065 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038498.517536_577.1","last_update":"2021-02-11T10:14:58.517536Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:02.065 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:02.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:02.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:02.066 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:02.108 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:02.108 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:02.360 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038498.517536_577.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:58.517536Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:02.360 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:02.360 INFO:tasks.rgw_multisite_tests:test s3 topics with secret set/get/delete on master ... ok 2021-02-11T10:15:02.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:02.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:02.361 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:02.402 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:02.402 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:02.639 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:02.684 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038498.517536_577.1","last_update":"2021-02-11T10:14:58.517536Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:02.684 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:15:02.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:02.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:02.685 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:02.727 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:02.727 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:03.004 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038498.517536_577.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:58.517536Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:03.004 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:15:03.004 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:15:03.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:03.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:03.005 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:03.045 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:03.045 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:03.319 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038498.517536_577.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:58.517536Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:03.320 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:15:03.320 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:03.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:03.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:03.320 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:03.360 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:03.360 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:03.598 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038498.517536_577.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:58.517536Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:03.598 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:03.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:03.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:03.599 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:03.866 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:03.910 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038498.517536_577.1","last_update":"2021-02-11T10:14:58.517536Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:03.910 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:03.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:03.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:03.911 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:03.951 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:03.951 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:04.209 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038498.517536_577.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:14:58.517536Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:04.210 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:05.298 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... ok 2021-02-11T10:15:05.299 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter_on_master ... ok 2021-02-11T10:15:05.299 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter ... ok 2021-02-11T10:15:05.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:05.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:05.299 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:05.339 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:05.339 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:05.558 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:05.599 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038505.236894_589.1","last_update":"2021-02-11T10:15:05.236894Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:05.600 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:15:05.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:05.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:05.600 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:05.643 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:05.643 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:05.898 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038505.236894_589.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:05.236894Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:05.899 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:15:05.899 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:15:05.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:05.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:05.899 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:05.943 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:05.943 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:06.209 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038505.236894_589.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:05.236894Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:06.209 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:15:06.209 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:06.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:06.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:06.210 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:06.250 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:06.251 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:06.501 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038505.236894_589.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:05.236894Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:06.501 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:06.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:06.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:06.502 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:06.540 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:06.541 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:06.786 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:06.822 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038505.236894_589.1","last_update":"2021-02-11T10:15:05.236894Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:06.823 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:06.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:06.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:06.823 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:06.863 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:06.863 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:07.118 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038505.236894_589.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:05.236894Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:07.118 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:08.196 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... ok 2021-02-11T10:15:08.197 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_objcet_timing ... ok 2021-02-11T10:15:08.197 INFO:tasks.rgw_multisite_tests:test pushing amqp s3 notification on master ... ok 2021-02-11T10:15:08.197 INFO:tasks.rgw_multisite_tests:test reservation cleanup after gateway crash ... ok 2021-02-11T10:15:08.197 INFO:tasks.rgw_multisite_tests:test gateway recovery of persistent notifications ... ok 2021-02-11T10:15:08.197 INFO:tasks.rgw_multisite_tests:test pushing persistent notification via two gateways ... ok 2021-02-11T10:15:08.198 INFO:tasks.rgw_multisite_tests:test pushing persistent notification when one of the endpoints has error ... ok 2021-02-11T10:15:08.198 INFO:tasks.rgw_multisite_tests:test pushing persistent notification http ... ok 2021-02-11T10:15:08.198 INFO:tasks.rgw_multisite_tests:test pushing persistent notification amqp ... ok 2021-02-11T10:15:08.198 INFO:tasks.rgw_multisite_tests:test pushing persistent notification of large notifications ... ok 2021-02-11T10:15:08.198 INFO:tasks.rgw_multisite_tests:test pushing persistent notification pushback ... ok 2021-02-11T10:15:08.198 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2021-02-11T10:15:08.199 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2021-02-11T10:15:08.199 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl ... ok 2021-02-11T10:15:08.199 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl_sasl ... ok 2021-02-11T10:15:08.199 INFO:tasks.rgw_multisite_tests:test deletion of multiple keys on master ... ok 2021-02-11T10:15:08.199 INFO:tasks.rgw_multisite_tests:test pushing http s3 notification on master ... ok 2021-02-11T10:15:08.199 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2021-02-11T10:15:08.200 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification on master ... ok 2021-02-11T10:15:08.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:08.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:08.200 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:08.499 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038508.181793_597.1","last_update":"2021-02-11T10:15:08.181793Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:08.499 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:15:08.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:08.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:08.500 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:08.540 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:08.540 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:08.806 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038508.181793_597.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:08.181793Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:08.806 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:15:08.807 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:15:08.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:08.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:08.807 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:08.845 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:08.845 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:09.116 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038508.181793_597.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:08.181793Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:09.116 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:15:09.116 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:09.116 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:09.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:09.117 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:09.158 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:09.158 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:09.392 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038508.181793_597.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:08.181793Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:09.393 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:09.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:09.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:09.393 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:09.433 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:09.433 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:09.646 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:09.685 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038508.181793_597.1","last_update":"2021-02-11T10:15:08.181793Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:09.685 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:09.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:09.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:09.685 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:09.726 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:09.726 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:09.959 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038508.181793_597.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:08.181793Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:09.959 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:09.980 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... ok 2021-02-11T10:15:09.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:09.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:09.981 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:10.021 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:10.021 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:10.279 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:10.324 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038508.181793_597.1","last_update":"2021-02-11T10:15:08.181793Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:10.324 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:15:10.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:10.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:10.325 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:10.366 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:10.366 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:10.608 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038508.181793_597.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:08.181793Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:10.608 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:15:10.608 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:15:10.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:10.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:10.609 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:10.651 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:10.915 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038508.181793_597.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:08.181793Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:10.916 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:15:10.916 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:10.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:10.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:10.916 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:10.954 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:10.954 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:11.194 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038508.181793_597.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:08.181793Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:11.194 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:11.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:11.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:11.195 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:11.234 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:11.234 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:11.464 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:11.508 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038508.181793_597.1","last_update":"2021-02-11T10:15:08.181793Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:11.509 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:11.509 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:11.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:11.509 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:11.550 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:11.550 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:11.767 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038508.181793_597.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:08.181793Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:11.767 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:11.786 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... ok 2021-02-11T10:15:11.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:11.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:11.786 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:11.825 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:11.825 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:12.049 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:12.094 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038508.181793_597.1","last_update":"2021-02-11T10:15:08.181793Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:12.094 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:15:12.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:12.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:12.095 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:12.135 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:12.135 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:12.415 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038508.181793_597.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:08.181793Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:12.415 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:15:12.415 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:15:12.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:12.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:12.416 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:12.456 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:12.456 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:12.739 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038508.181793_597.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:08.181793Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:12.739 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:15:12.739 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:12.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:12.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:12.739 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:12.781 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:12.781 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:13.032 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038508.181793_597.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:08.181793Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:13.032 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:13.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:13.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:13.032 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:13.073 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:13.073 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:13.302 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:13.346 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038508.181793_597.1","last_update":"2021-02-11T10:15:08.181793Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:13.346 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:13.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:13.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:13.347 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:13.388 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:13.388 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:13.620 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038508.181793_597.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:08.181793Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:13.621 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:13.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:13.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:13.809 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:13.846 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:13.846 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:14.067 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:14.110 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038513.803016_607.1","last_update":"2021-02-11T10:15:13.803016Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:14.110 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:14.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:14.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:14.111 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:14.151 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:14.152 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:14.401 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038513.803016_607.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:13.803016Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:14.401 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:14.546 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... ok 2021-02-11T10:15:14.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:14.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:14.546 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:14.589 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:14.589 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:14.802 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:14.839 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038514.530223_610.1","last_update":"2021-02-11T10:15:14.530223Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:14.840 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:15:14.840 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:14.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:14.840 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:14.880 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:14.880 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:15.142 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038514.530223_610.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:14.530223Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:15.142 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:15:15.142 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:15:15.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:15.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:15.143 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:15.185 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:15.186 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:15.454 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038514.530223_610.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:14.530223Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:15.455 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:15:15.455 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:15.455 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:15.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:15.455 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:15.495 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:15.495 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:15.746 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038514.530223_610.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:14.530223Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:15.746 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:15.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:15.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:15.747 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:15.788 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:15.788 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:16.010 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:16.054 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038514.530223_610.1","last_update":"2021-02-11T10:15:14.530223Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:16.054 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:16.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:16.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:16.054 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:16.096 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:16.096 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:16.343 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038514.530223_610.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:14.530223Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:16.344 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:16.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:16.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:16.494 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:16.530 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:16.530 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:16.774 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038516.489292_616.1","last_update":"2021-02-11T10:15:16.489292Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:16.774 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:16.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:16.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:16.775 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:16.816 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:16.816 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:17.065 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038516.489292_616.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:16.489292Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:17.066 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:17.216 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... ok 2021-02-11T10:15:17.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:17.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:17.217 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:17.259 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:17.259 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:17.487 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:17.523 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038517.201822_619.1","last_update":"2021-02-11T10:15:17.201822Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:17.524 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:15:17.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:17.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:17.524 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:17.565 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:17.565 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:17.821 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038517.201822_619.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:17.201822Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:17.821 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:15:17.821 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:15:17.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:17.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:17.821 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:17.861 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:17.861 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:18.119 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038517.201822_619.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:17.201822Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:18.119 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:15:18.119 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:18.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:18.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:18.120 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:18.159 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:18.159 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:18.394 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038517.201822_619.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:17.201822Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:18.394 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:18.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:18.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:18.395 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:18.435 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:18.435 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:18.667 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:18.714 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038517.201822_619.1","last_update":"2021-02-11T10:15:17.201822Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:18.715 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:18.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:18.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:18.715 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:18.753 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:18.754 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:19.007 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038517.201822_619.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:17.201822Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:19.008 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:19.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:19.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:19.203 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:19.238 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:19.238 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:19.451 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:19.490 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038519.198339_625.1","last_update":"2021-02-11T10:15:19.198339Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:19.490 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:19.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:19.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:19.491 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:19.532 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:19.648 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:19.777 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038519.198339_625.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:19.198339Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:19.777 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:20.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:20.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:20.223 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-84 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:15:20.272 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:20.272 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:20.508 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:20.507+0000 7f4e2c60c080 1 bucket sync caught up with source: 2021-02-11T10:15:20.508 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000002.200.5, , 00000000001.70.5, 00000000001.95.5, 00000000001.95.5, , 00000000001.110.5, 00000000001.76.5, 00000000001.92.5, 00000000002.118.5] 2021-02-11T10:15:20.508 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000002.200.5, , 00000000001.70.5, 00000000001.95.5, 00000000001.95.5, , 00000000001.110.5, 00000000001.76.5, 00000000001.92.5, 00000000002.118.5] 2021-02-11T10:15:20.508 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:20.507+0000 7f4e2c60c080 0 bucket checkpoint complete 2021-02-11T10:15:20.545 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "0" type: "OBJECT_CREATE" 2021-02-11T10:15:20.545 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "5" type: "OBJECT_CREATE" 2021-02-11T10:15:20.545 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "2" type: "OBJECT_CREATE" 2021-02-11T10:15:20.545 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "4" type: "OBJECT_CREATE" 2021-02-11T10:15:20.545 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "3" type: "OBJECT_CREATE" 2021-02-11T10:15:20.545 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "1" type: "OBJECT_CREATE" 2021-02-11T10:15:20.545 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "9" type: "OBJECT_CREATE" 2021-02-11T10:15:20.545 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "6" type: "OBJECT_CREATE" 2021-02-11T10:15:20.546 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "7" type: "OBJECT_CREATE" 2021-02-11T10:15:20.546 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2021-02-11T10:15:20.554 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T10:15:20.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:20.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:20.666 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-84 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:15:20.718 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:20.718 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:20.932 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:20.931+0000 7fc81031e080 1 bucket sync caught up with source: 2021-02-11T10:15:20.932 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000004.204.5, , 00000000002.72.5, 00000000002.97.5, 00000000002.97.5, , 00000000002.112.5, 00000000002.78.5, 00000000002.94.5, 00000000004.122.5] 2021-02-11T10:15:20.933 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000004.204.5, , 00000000002.72.5, 00000000002.97.5, 00000000002.97.5, , 00000000002.112.5, 00000000002.78.5, 00000000002.94.5, 00000000004.122.5] 2021-02-11T10:15:20.933 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:20.931+0000 7fc81031e080 0 bucket checkpoint complete 2021-02-11T10:15:21.090 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... ok 2021-02-11T10:15:21.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:21.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:21.091 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:21.128 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:21.128 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:21.365 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:21.410 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038521.078490_632.1","last_update":"2021-02-11T10:15:21.078490Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:21.411 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:15:21.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:21.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:21.411 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:21.452 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:21.452 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:21.703 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038521.078490_632.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:21.078490Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:21.703 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:15:21.703 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:15:21.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:21.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:21.704 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:21.745 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:21.745 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:21.994 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038521.078490_632.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:21.078490Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:21.995 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:15:21.995 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:21.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:21.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:21.996 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:22.035 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:22.035 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:22.279 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038521.078490_632.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:21.078490Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:22.279 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:22.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:22.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:22.280 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:22.323 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:22.323 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:22.550 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:22.591 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038521.078490_632.1","last_update":"2021-02-11T10:15:21.078490Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:22.591 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:22.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:22.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:22.592 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:22.630 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:22.630 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:22.869 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038521.078490_632.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:21.078490Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:22.870 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:22.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:15:22.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:15:22.879 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 topic list --uid rgw-multisite-test-user --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:15:22.951 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:22.951 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:23.209 INFO:teuthology.orchestra.run.smithi133.stdout:{"topics":[{"topic":{"user":"","name":"erivkc-85_topic","dest":{"bucket_name":"","oid_prefix":"","push_endpoint":"","push_endpoint_args":"","push_endpoint_topic":"erivkc-85_topic","stored_secret":"false","persistent":"false"},"arn":"arn:aws:sns:test-zonegroup::erivkc-85_topic","opaqueData":""},"subs":[]}]} 2021-02-11T10:15:23.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'erivkc-85_topic', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:15:23.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'erivkc-85_topic', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:15:23.210 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 topic get --uid rgw-multisite-test-user --topic erivkc-85_topic --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:15:23.291 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:23.291 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:23.538 INFO:teuthology.orchestra.run.smithi133.stdout:{"topic":{"user":"","name":"erivkc-85_topic","dest":{"bucket_name":"","oid_prefix":"","push_endpoint":"","push_endpoint_args":"","push_endpoint_topic":"erivkc-85_topic","stored_secret":"false","persistent":"false"},"arn":"arn:aws:sns:test-zonegroup::erivkc-85_topic","opaqueData":""},"subs":[]} 2021-02-11T10:15:23.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:15:23.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:15:23.559 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 topic list --uid rgw-multisite-test-user --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:15:23.597 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:23.597 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:23.853 INFO:teuthology.orchestra.run.smithi026.stdout:{"topics":[{"topic":{"user":"","name":"erivkc-85_topic","dest":{"bucket_name":"","oid_prefix":"","push_endpoint":"amqp://127.0.0.1:7001/vhost_1","push_endpoint_args":"Attributes.entry.1.key=push-endpoint&Attributes.entry.1.value=amqp://127.0.0.1:7001/vhost_1&Attributes.entry.2.key=amqp-exchange&Attributes.entry.2.value=amqp.direct&Attributes.entry.3.key=amqp-ack-level&Attributes.entry.3.value=none&Version=2010-03-31&amqp-ack-level=none&amqp-exchange=amqp.direct&push-endpoint=amqp://127.0.0.1:7001/vhost_1","push_endpoint_topic":"erivkc-85_topic","stored_secret":"false","persistent":"false"},"arn":"arn:aws:sns:test-zonegroup::erivkc-85_topic","opaqueData":""},"subs":[]}]} 2021-02-11T10:15:23.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'erivkc-85_topic', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:15:23.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'erivkc-85_topic', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:15:23.854 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 topic get --uid rgw-multisite-test-user --topic erivkc-85_topic --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:15:23.934 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:23.935 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:24.198 INFO:teuthology.orchestra.run.smithi026.stdout:{"topic":{"user":"","name":"erivkc-85_topic","dest":{"bucket_name":"","oid_prefix":"","push_endpoint":"amqp://127.0.0.1:7001/vhost_1","push_endpoint_args":"Attributes.entry.1.key=push-endpoint&Attributes.entry.1.value=amqp://127.0.0.1:7001/vhost_1&Attributes.entry.2.key=amqp-exchange&Attributes.entry.2.value=amqp.direct&Attributes.entry.3.key=amqp-ack-level&Attributes.entry.3.value=none&Version=2010-03-31&amqp-ack-level=none&amqp-exchange=amqp.direct&push-endpoint=amqp://127.0.0.1:7001/vhost_1","push_endpoint_topic":"erivkc-85_topic","stored_secret":"false","persistent":"false"},"arn":"arn:aws:sns:test-zonegroup::erivkc-85_topic","opaqueData":""},"subs":[]} 2021-02-11T10:15:24.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:24.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:24.475 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:24.510 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:24.510 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:24.722 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:24.764 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038524.469357_640.1","last_update":"2021-02-11T10:15:24.469357Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:24.764 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:24.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:24.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:24.765 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:24.805 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:24.806 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:25.044 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038524.469357_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:24.469357Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:25.044 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:25.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'get', '--uid', 'rgw-multisite-test-user', '--subscription', 'erivkc-85_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:15:25.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'get', '--uid', 'rgw-multisite-test-user', '--subscription', 'erivkc-85_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:15:25.066 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription get --uid rgw-multisite-test-user --subscription erivkc-85_sub --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:15:25.126 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:25.126 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:25.383 INFO:teuthology.orchestra.run.smithi133.stdout:{"user":"","name":"erivkc-85_sub","topic":"erivkc-85_topic","dest":{"bucket_name":"pubsub-rgw-multisite-test-user-erivkc-85_topic","oid_prefix":"erivkc-85_sub/","push_endpoint":"","push_endpoint_args":"topic=erivkc-85_topic","push_endpoint_topic":"erivkc-85_topic","stored_secret":"false","persistent":"false"},"s3_id":""} 2021-02-11T10:15:30.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-85', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:30.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-85', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:30.228 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-85 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:15:30.276 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:30.276 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:30.496 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:30.494+0000 7fd36c566080 1 bucket sync caught up with source: 2021-02-11T10:15:30.496 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000011.92.5, 00000000011.136.5, 00000000010.93.5, 00000000010.225.5, 00000000009.103.5, 00000000008.76.5, 00000000010.87.5, 00000000010.133.5, 00000000011.80.5, 00000000011.122.5, 00000000010.118.5] 2021-02-11T10:15:30.496 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000011.92.5, 00000000011.136.5, 00000000010.93.5, 00000000010.225.5, 00000000009.103.5, 00000000008.76.5, 00000000010.87.5, 00000000010.133.5, 00000000011.80.5, 00000000011.122.5, 00000000010.118.5] 2021-02-11T10:15:30.496 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:30.494+0000 7fd36c566080 0 bucket checkpoint complete 2021-02-11T10:15:30.532 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'erivkc-85_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:15:30.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'erivkc-85_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:15:30.533 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription pull --uid rgw-multisite-test-user --subscription erivkc-85_sub --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:15:30.573 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:30.573 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:30.835 INFO:teuthology.orchestra.run.smithi133.stdout:{"next_marker":"erivkc-85_sub/1613038529.844356.da260415","is_truncated":"true","events":[{"id":"1613038525.497548.d9fbdcc0","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:25.497548Z","info":{"attrs":{"mtime":"2021-02-11T10:15:25.386873Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"0"}}},{"id":"1613038525.580107.c25c0014","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:25.580107Z","info":{"attrs":{"mtime":"2021-02-11T10:15:25.406026Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"1"}}},{"id":"1613038525.696142.546684c5","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:25.696142Z","info":{"attrs":{"mtime":"2021-02-11T10:15:25.580528Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"5"}}},{"id":"1613038525.740151.c5f821ee","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:25.740151Z","info":{"attrs":{"mtime":"2021-02-11T10:15:25.448610Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"2"}}},{"id":"1613038525.845795.6c9b1f3a","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:25.845795Z","info":{"attrs":{"mtime":"2021-02-11T10:15:25.756922Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"9"}}},{"id":"1613038525.895510.6a35e452","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:25.895510Z","info":{"attrs":{"mtime":"2021-02-11T10:15:25.801564Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"10"}}},{"id":"1613038525.896777.2f4dab56","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:25.896777Z","info":{"attrs":{"mtime":"2021-02-11T10:15:25.492630Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"3"}}},{"id":"1613038525.903783.545557d5","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:25.903783Z","info":{"attrs":{"mtime":"2021-02-11T10:15:25.712317Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"8"}}},{"id":"1613038525.952385.8081043d","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:25.952385Z","info":{"attrs":{"mtime":"2021-02-11T10:15:25.536761Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"4"}}},{"id":"1613038525.952641.3d987df3","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:25.952641Z","info":{"attrs":{"mtime":"2021-02-11T10:15:25.668616Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"7"}}},{"id":"1613038525.975870.83dca72a","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:25.975870Z","info":{"attrs":{"mtime":"2021-02-11T10:15:25.625217Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"6"}}},{"id":"1613038525.994389.2ef954b4","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:25.994389Z","info":{"attrs":{"mtime":"2021-02-11T10:15:25.888514Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"12"}}},{"id":"1613038526.042301.5e6c7834","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.042301Z","info":{"attrs":{"mtime":"2021-02-11T10:15:25.932462Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"13"}}},{"id":"1613038526.043826.8885ebc3","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.043826Z","info":{"attrs":{"mtime":"2021-02-11T10:15:25.844354Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"11"}}},{"id":"1613038526.059917.2ef954b4","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.059917Z","info":{"attrs":{"mtime":"2021-02-11T10:15:25.888514Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"12"}}},{"id":"1613038526.242688.a32029e5","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.242688Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.108585Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"17"}}},{"id":"1613038526.243778.3ccba508","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.243778Z","info":{"attrs":{"mtime":"2021-02-11T10:15:25.976204Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"14"}}},{"id":"1613038526.246867.53493e3a","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.246867Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.196418Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"19"}}},{"id":"1613038526.282329.73f2bf19","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.282329Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.064448Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"16"}}},{"id":"1613038526.287892.17a8f4cd","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.287892Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.152288Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"18"}}},{"id":"1613038526.331876.0e34154a","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.331876Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.020328Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"15"}}},{"id":"1613038526.444045.7c2a20b5","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.444045Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.284215Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"21"}}},{"id":"1613038526.448333.6e86bc77","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.448333Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.372538Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"23"}}},{"id":"1613038526.492109.efcad98f","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.492109Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.328596Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"22"}}},{"id":"1613038526.535996.e1b78109","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.535996Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.240203Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"20"}}},{"id":"1613038526.587926.d5371e05","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.587926Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.416914Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"24"}}},{"id":"1613038526.649551.cc9dd9dc","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.649551Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.592584Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"28"}}},{"id":"1613038526.684027.d339616f","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.684027Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.460391Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"25"}}},{"id":"1613038526.690321.0aa9b28b","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.690321Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.504434Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"26"}}},{"id":"1613038526.728166.75381355","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.728166Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.548382Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"27"}}},{"id":"1613038526.846314.8b3fb03b","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.846314Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.636691Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"29"}}},{"id":"1613038526.846625.235f5d4c","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.846625Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.724491Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"31"}}},{"id":"1613038526.891083.6b24cfd6","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.891083Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.768597Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"32"}}},{"id":"1613038526.932069.d8e0a4e5","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.932069Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.812974Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"33"}}},{"id":"1613038526.944528.82c5bcfe","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:26.944528Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.683031Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"30"}}},{"id":"1613038527.047273.65a43df4","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.047273Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.988555Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"37"}}},{"id":"1613038527.048334.27af5078","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.048334Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.944796Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"36"}}},{"id":"1613038527.051063.6318360a","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.051063Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.856291Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"34"}}},{"id":"1613038527.140048.d701c196","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.140048Z","info":{"attrs":{"mtime":"2021-02-11T10:15:26.900391Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"35"}}},{"id":"1613038527.240929.61fc286a","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.240929Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.033125Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"38"}}},{"id":"1613038527.242307.c5f7ef67","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.242307Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.164423Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"41"}}},{"id":"1613038527.289119.8ed4df45","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.289119Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.076428Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"39"}}},{"id":"1613038527.331952.a1614078","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.331952Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.208826Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"42"}}},{"id":"1613038527.342639.d01cd557","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.342639Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.120393Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"40"}}},{"id":"1613038527.441837.789ccd30","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.441837Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.296540Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"44"}}},{"id":"1613038527.486687.b2ec39a5","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.486687Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.385143Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"46"}}},{"id":"1613038527.487249.12ff5292","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.487249Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.340866Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"45"}}},{"id":"1613038527.496338.65b1c8b2","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.496338Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.252429Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"43"}}},{"id":"1613038527.548141.40e6ae6f","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.548141Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.428762Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"47"}}},{"id":"1613038527.645306.d4e729bc","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.645306Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.516304Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"49"}}},{"id":"1613038527.646009.5c27a6dd","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.646009Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.560431Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"50"}}},{"id":"1613038527.647091.de9c8c94","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.647091Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.472268Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"48"}}},{"id":"1613038527.648784.1f41c2b9","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.648784Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.604471Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"51"}}},{"id":"1613038527.844276.1cb795e8","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.844276Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.736531Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"54"}}},{"id":"1613038527.847689.15bf206a","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.847689Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.648696Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"52"}}},{"id":"1613038527.894347.87ef0538","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.894347Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.692525Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"53"}}},{"id":"1613038527.927904.2ba4f024","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:27.927904Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.780514Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"55"}}},{"id":"1613038528.045385.29b68681","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.045385Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.912499Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"58"}}},{"id":"1613038528.045708.85eeeb87","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.045708Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.956555Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"59"}}},{"id":"1613038528.046251.2da63430","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.046251Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.868363Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"57"}}},{"id":"1613038528.089231.18fcd76c","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.089231Z","info":{"attrs":{"mtime":"2021-02-11T10:15:27.824711Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"56"}}},{"id":"1613038528.136069.d3f4ded2","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.136069Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.000812Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"60"}}},{"id":"1613038528.243840.667602dd","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.243840Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.132560Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"63"}}},{"id":"1613038528.290524.d52c7213","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.290524Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.176486Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"64"}}},{"id":"1613038528.294357.35d2de51","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.294357Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.088363Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"62"}}},{"id":"1613038528.294825.a4af40d2","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.294825Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.220499Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"65"}}},{"id":"1613038528.297873.ecf0bfa6","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.297873Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.044383Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"61"}}},{"id":"1613038528.353601.f62218ac","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.353601Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.268481Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"66"}}},{"id":"1613038528.441668.49299c1a","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.441668Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.408771Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"69"}}},{"id":"1613038528.485009.44359bbd","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.485009Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.316420Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"67"}}},{"id":"1613038528.530420.8dd8c6b5","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.530420Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.452492Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"70"}}},{"id":"1613038528.532183.2c8afc76","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.532183Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.364320Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"68"}}},{"id":"1613038528.644122.f39d5903","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.644122Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.584729Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"73"}}},{"id":"1613038528.693263.0d8c6514","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.693263Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.497055Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"71"}}},{"id":"1613038528.732008.da587fb9","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.732008Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.540837Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"72"}}},{"id":"1613038528.844125.490a5408","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.844125Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.672622Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"75"}}},{"id":"1613038528.844957.0f417054","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.844957Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.716464Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"76"}}},{"id":"1613038528.928039.95e9d1cf","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.928039Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.628920Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"74"}}},{"id":"1613038528.928184.7e9f410f","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:28.928184Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.764469Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"77"}}},{"id":"1613038529.047002.9a002b36","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.047002Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.904582Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"80"}}},{"id":"1613038529.092732.b7607832","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.092732Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.812449Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"78"}}},{"id":"1613038529.093243.c17c9304","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.093243Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.948642Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"81"}}},{"id":"1613038529.097427.5569604e","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.097427Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.860624Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"79"}}},{"id":"1613038529.101252.4d73c741","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.101252Z","info":{"attrs":{"mtime":"2021-02-11T10:15:28.992714Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"82"}}},{"id":"1613038529.243960.b77ebd9c","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.243960Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.216905Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"87"}}},{"id":"1613038529.244270.9809ca56","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.244270Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.128373Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"85"}}},{"id":"1613038529.283283.02c834dd","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.283283Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.084626Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"84"}}},{"id":"1613038529.292368.1902826a","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.292368Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.039574Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"83"}}},{"id":"1613038529.336222.ead29ae5","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.336222Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.172673Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"86"}}},{"id":"1613038529.442292.4a8c6995","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.442292Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.348382Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"90"}}},{"id":"1613038529.488333.06d6a1f9","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.488333Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.304316Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"89"}}},{"id":"1613038529.492382.56be5ee7","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.492382Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.260409Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"88"}}},{"id":"1613038529.492789.4d8d6a8c","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.492789Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.392708Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"91"}}},{"id":"1613038529.642672.e94ea46e","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.642672Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.524674Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"94"}}},{"id":"1613038529.642981.5f8be16e","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.642981Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.612729Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"96"}}},{"id":"1613038529.645013.5eb9a434","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.645013Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.436551Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"92"}}},{"id":"1613038529.645458.35a8f194","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.645458Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.568356Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"95"}}},{"id":"1613038529.688359.b6839b0a","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.688359Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.480815Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"93"}}},{"id":"1613038529.737827.b4777bbe","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.737827Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.656424Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"97"}}},{"id":"1613038529.844356.da260415","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.844356Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.788493Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"100"}}}]} 2021-02-11T10:15:30.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'erivkc-85_sub', '--marker', 'erivkc-85_sub/1613038529.844356.da260415', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:15:30.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'erivkc-85_sub', '--marker', 'erivkc-85_sub/1613038529.844356.da260415', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:15:30.836 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription pull --uid rgw-multisite-test-user --subscription erivkc-85_sub --marker erivkc-85_sub/1613038529.844356.da260415 --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:15:30.919 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:30.919 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:31.167 INFO:teuthology.orchestra.run.smithi133.stdout:{"next_marker":"","is_truncated":"false","events":[{"id":"1613038529.848056.8df5e786","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.848056Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.700243Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"98"}}},{"id":"1613038529.888125.d8032acb","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:29.888125Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.744415Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"99"}}},{"id":"1613038530.045045.818516ee","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:30.045045Z","info":{"attrs":{"mtime":"2021-02-11T10:15:30.008691Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"105"}}},{"id":"1613038530.047294.db5aacd4","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:30.047294Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.920502Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"103"}}},{"id":"1613038530.093749.698f9977","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:30.093749Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.964472Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"104"}}},{"id":"1613038530.094270.3590a7bd","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:30.094270Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.833003Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"101"}}},{"id":"1613038530.099968.fded09de","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:30.099968Z","info":{"attrs":{"mtime":"2021-02-11T10:15:29.876390Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"102"}}},{"id":"1613038530.245795.6754319c","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:30.245795Z","info":{"attrs":{"mtime":"2021-02-11T10:15:30.184536Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"109"}}},{"id":"1613038530.292894.91aa7b35","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:30.292894Z","info":{"attrs":{"mtime":"2021-02-11T10:15:30.140485Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"108"}}},{"id":"1613038530.296568.9b24ac0c","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:30.296568Z","info":{"attrs":{"mtime":"2021-02-11T10:15:30.052369Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"106"}}},{"id":"1613038530.297165.3e5c89b2","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:15:30.297165Z","info":{"attrs":{"mtime":"2021-02-11T10:15:30.096369Z"},"bucket":{"bucket_id":"a3c4910a-4529-4294-aed4-ef409f209cf5.5479.33","name":"erivkc-85","tenant":""},"key":{"instance":"","name":"107"}}}]} 2021-02-11T10:15:31.172 INFO:tasks.rgw_multisite_tests:test radosgw-admin commands ... FAIL 2021-02-11T10:15:31.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:31.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:31.173 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:31.439 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:31.486 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038524.469357_640.1","last_update":"2021-02-11T10:15:24.469357Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:31.486 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:15:31.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:31.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:31.486 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:31.526 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:31.526 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:31.780 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038524.469357_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:24.469357Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:31.781 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:15:31.781 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:15:31.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:31.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:31.781 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:32.076 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038524.469357_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:24.469357Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:32.076 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:15:32.077 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:32.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:32.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:32.077 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:32.118 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:32.118 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:32.361 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038524.469357_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:24.469357Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:32.362 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:32.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:32.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:32.362 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:32.403 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:32.403 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:32.648 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:32.688 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038524.469357_640.1","last_update":"2021-02-11T10:15:24.469357Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:32.688 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:32.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:32.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:32.688 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:32.727 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:32.727 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:32.992 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038524.469357_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:24.469357Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:32.992 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:34.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-86', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:34.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-86', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:34.495 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-86 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:15:34.537 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:34.537 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:34.772 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:34.770+0000 7f27952a2080 1 waiting for incremental sync to catch up: 2021-02-11T10:15:34.772 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , 00000000001.100.5, 00000000001.64.5, 00000000001.87.5, , 00000000001.96.5, 00000000001.89.5, 00000000001.122.5, 00000000002.85.5] 2021-02-11T10:15:34.772 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000002.230.5, , 00000000001.100.5, 00000000001.64.5, 00000000001.87.5, , 00000000001.96.5, 00000000001.89.5, 00000000001.122.5, 00000000002.85.5] 2021-02-11T10:15:39.776 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:39.774+0000 7f27952a2080 1 waiting for incremental sync to catch up: 2021-02-11T10:15:39.777 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , 00000000001.100.5, 00000000001.64.5, 00000000001.87.5, , 00000000001.96.5, 00000000001.89.5, 00000000001.122.5, 00000000002.85.5] 2021-02-11T10:15:39.777 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000002.230.5, , 00000000001.100.5, 00000000001.64.5, 00000000001.87.5, , 00000000001.96.5, 00000000001.89.5, 00000000001.122.5, 00000000002.85.5] 2021-02-11T10:15:44.780 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:44.778+0000 7f27952a2080 1 bucket sync caught up with source: 2021-02-11T10:15:44.780 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000002.230.5, , 00000000001.100.5, 00000000001.64.5, 00000000001.87.5, , 00000000001.96.5, 00000000001.89.5, 00000000001.122.5, 00000000002.85.5] 2021-02-11T10:15:44.781 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000002.230.5, , 00000000001.100.5, 00000000001.64.5, 00000000001.87.5, , 00000000001.96.5, 00000000001.89.5, 00000000001.122.5, 00000000002.85.5] 2021-02-11T10:15:44.781 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:44.778+0000 7f27952a2080 0 bucket checkpoint complete 2021-02-11T10:15:45.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-86', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:45.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-86', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:45.695 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-86 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:15:45.741 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:45.741 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:45.942 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:45.942+0000 7f4fce198080 1 waiting for incremental sync to catch up: 2021-02-11T10:15:45.942 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.79.5, 00000000003.232.5, , 00000000002.102.5, 00000000001.64.5, 00000000002.91.5, , 00000000003.100.5, 00000000001.89.5, 00000000001.122.5, 00000000003.87.5] 2021-02-11T10:15:45.942 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.79.5, 00000000003.232.5, , 00000000002.102.5, 00000000001.64.5, 00000000003.93.5, , 00000000003.100.5, 00000000002.95.5, 00000000002.124.5, 00000000003.87.5] 2021-02-11T10:15:50.947 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:50.946+0000 7f4fce198080 1 bucket sync caught up with source: 2021-02-11T10:15:50.947 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.79.5, 00000000003.232.5, , 00000000002.102.5, 00000000001.64.5, 00000000003.93.5, , 00000000003.100.5, 00000000002.95.5, 00000000002.124.5, 00000000003.87.5] 2021-02-11T10:15:50.947 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.79.5, 00000000003.232.5, , 00000000002.102.5, 00000000001.64.5, 00000000003.93.5, , 00000000003.100.5, 00000000002.95.5, 00000000002.124.5, 00000000003.87.5] 2021-02-11T10:15:50.947 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:50.946+0000 7f4fce198080 0 bucket checkpoint complete 2021-02-11T10:15:50.993 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "12" type: "OBJECT_CREATE" 2021-02-11T10:15:50.993 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "13" type: "OBJECT_CREATE" 2021-02-11T10:15:50.993 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "11" type: "OBJECT_CREATE" 2021-02-11T10:15:50.993 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "15" type: "OBJECT_CREATE" 2021-02-11T10:15:50.993 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "10" type: "OBJECT_CREATE" 2021-02-11T10:15:50.993 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "18" type: "OBJECT_CREATE" 2021-02-11T10:15:50.993 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "17" type: "OBJECT_CREATE" 2021-02-11T10:15:50.993 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "14" type: "OBJECT_CREATE" 2021-02-11T10:15:50.993 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "16" type: "OBJECT_CREATE" 2021-02-11T10:15:50.993 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "19" type: "OBJECT_CREATE" 2021-02-11T10:15:51.941 INFO:tasks.rgw_multisite_tests:test that events are only sent on incremental sync ... ok 2021-02-11T10:15:51.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:51.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:51.942 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:51.980 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:51.980 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:52.167 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:52.207 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038551.922129_652.1","last_update":"2021-02-11T10:15:51.922129Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:52.208 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:15:52.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:52.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:52.208 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:52.249 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:52.250 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:52.512 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038551.922129_652.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:51.922129Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:52.513 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:15:52.513 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:15:52.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:52.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:52.513 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:52.554 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:52.554 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:52.812 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038551.922129_652.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:51.922129Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:52.812 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:15:52.812 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:52.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:52.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:52.813 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:52.853 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:52.853 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:53.105 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038551.922129_652.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:51.922129Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:53.105 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:53.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:53.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:53.105 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:53.147 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:53.147 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:53.372 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:53.415 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038551.922129_652.1","last_update":"2021-02-11T10:15:51.922129Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:53.415 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:53.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:53.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:53.416 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:53.458 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:53.458 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:53.693 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038551.922129_652.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:51.922129Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:53.693 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:54.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:54.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:54.283 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:54.318 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:54.318 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:54.536 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:54.582 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038554.277519_659.1","last_update":"2021-02-11T10:15:54.277519Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:54.582 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:54.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:54.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:54.583 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:54.621 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:54.621 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:54.867 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038554.277519_659.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:54.277519Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:54.868 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:54.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-87', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:54.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-87', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:54.868 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-87 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:15:54.956 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:54.956 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:55.184 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:55.182+0000 7fc466e03080 1 bucket sync caught up with empty source 2021-02-11T10:15:55.185 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:55.182+0000 7fc466e03080 0 bucket checkpoint complete 2021-02-11T10:15:55.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-87', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:55.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-87', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:55.763 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-87 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:15:55.819 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:55.819 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:56.034 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:56.034+0000 7fb1ed772080 1 bucket sync caught up with source: 2021-02-11T10:15:56.034 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000002.103.5, , 00000000001.95.5, 00000000001.103.5, 00000000001.84.5, , 00000000001.110.5, 00000000001.119.5, 00000000001.109.5, 00000000002.83.5] 2021-02-11T10:15:56.034 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000002.103.5, , 00000000001.95.5, 00000000001.103.5, 00000000001.84.5, , 00000000001.110.5, 00000000001.119.5, 00000000001.109.5, 00000000002.83.5] 2021-02-11T10:15:56.034 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:56.034+0000 7fb1ed772080 0 bucket checkpoint complete 2021-02-11T10:15:56.073 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "2" type: "OBJECT_CREATE" 2021-02-11T10:15:56.073 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "1" type: "OBJECT_CREATE" 2021-02-11T10:15:56.073 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "3" type: "OBJECT_CREATE" 2021-02-11T10:15:56.073 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "0" type: "OBJECT_CREATE" 2021-02-11T10:15:56.073 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "6" type: "OBJECT_CREATE" 2021-02-11T10:15:56.073 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "5" type: "OBJECT_CREATE" 2021-02-11T10:15:56.073 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "7" type: "OBJECT_CREATE" 2021-02-11T10:15:56.073 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "9" type: "OBJECT_CREATE" 2021-02-11T10:15:56.073 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "4" type: "OBJECT_CREATE" 2021-02-11T10:15:56.074 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "8" type: "OBJECT_CREATE" 2021-02-11T10:15:56.083 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T10:15:56.094 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "2" type: "OBJECT_CREATE" 2021-02-11T10:15:56.094 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "1" type: "OBJECT_CREATE" 2021-02-11T10:15:56.094 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "3" type: "OBJECT_CREATE" 2021-02-11T10:15:56.094 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "0" type: "OBJECT_CREATE" 2021-02-11T10:15:56.094 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "6" type: "OBJECT_CREATE" 2021-02-11T10:15:56.095 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "5" type: "OBJECT_CREATE" 2021-02-11T10:15:56.095 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "7" type: "OBJECT_CREATE" 2021-02-11T10:15:56.095 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "9" type: "OBJECT_CREATE" 2021-02-11T10:15:56.095 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "4" type: "OBJECT_CREATE" 2021-02-11T10:15:56.095 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "8" type: "OBJECT_CREATE" 2021-02-11T10:15:56.095 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T10:15:56.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-87', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:56.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-87', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:56.186 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-87 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:15:56.237 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:56.237 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:56.491 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:56.490+0000 7f1b5719b080 1 bucket sync caught up with source: 2021-02-11T10:15:56.492 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000004.107.5, , 00000000002.97.5, 00000000002.105.5, 00000000002.86.5, , 00000000002.112.5, 00000000002.121.5, 00000000002.111.5, 00000000004.87.5] 2021-02-11T10:15:56.492 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000004.107.5, , 00000000002.97.5, 00000000002.105.5, 00000000002.86.5, , 00000000002.112.5, 00000000002.121.5, 00000000002.111.5, 00000000004.87.5] 2021-02-11T10:15:56.492 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:15:56.490+0000 7f1b5719b080 0 bucket checkpoint complete 2021-02-11T10:15:56.523 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE) synced 2021-02-11T10:15:56.529 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "2" type: "OBJECT_CREATE" 2021-02-11T10:15:56.529 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "1" type: "OBJECT_CREATE" 2021-02-11T10:15:56.529 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "3" type: "OBJECT_CREATE" 2021-02-11T10:15:56.529 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "0" type: "OBJECT_CREATE" 2021-02-11T10:15:56.529 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "6" type: "OBJECT_CREATE" 2021-02-11T10:15:56.529 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "5" type: "OBJECT_CREATE" 2021-02-11T10:15:56.530 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "7" type: "OBJECT_CREATE" 2021-02-11T10:15:56.530 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "9" type: "OBJECT_CREATE" 2021-02-11T10:15:56.530 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "4" type: "OBJECT_CREATE" 2021-02-11T10:15:56.530 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "8" type: "OBJECT_CREATE" 2021-02-11T10:15:56.530 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T10:15:56.537 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "2" type: "OBJECT_DELETE" 2021-02-11T10:15:56.537 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "1" type: "OBJECT_DELETE" 2021-02-11T10:15:56.537 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "6" type: "OBJECT_DELETE" 2021-02-11T10:15:56.537 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "0" type: "OBJECT_DELETE" 2021-02-11T10:15:56.537 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "3" type: "OBJECT_DELETE" 2021-02-11T10:15:56.537 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "7" type: "OBJECT_DELETE" 2021-02-11T10:15:56.537 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "5" type: "OBJECT_DELETE" 2021-02-11T10:15:56.537 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "9" type: "OBJECT_DELETE" 2021-02-11T10:15:56.537 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "4" type: "OBJECT_DELETE" 2021-02-11T10:15:56.537 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "8" type: "OBJECT_DELETE" 2021-02-11T10:15:56.537 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T10:15:56.544 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "2" type: "OBJECT_CREATE" 2021-02-11T10:15:56.544 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "1" type: "OBJECT_CREATE" 2021-02-11T10:15:56.544 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "3" type: "OBJECT_CREATE" 2021-02-11T10:15:56.544 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "0" type: "OBJECT_CREATE" 2021-02-11T10:15:56.544 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "6" type: "OBJECT_CREATE" 2021-02-11T10:15:56.544 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "5" type: "OBJECT_CREATE" 2021-02-11T10:15:56.544 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "7" type: "OBJECT_CREATE" 2021-02-11T10:15:56.544 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "9" type: "OBJECT_CREATE" 2021-02-11T10:15:56.544 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "4" type: "OBJECT_CREATE" 2021-02-11T10:15:56.545 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "8" type: "OBJECT_CREATE" 2021-02-11T10:15:56.545 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T10:15:56.730 INFO:tasks.rgw_multisite_tests:test subscriptions for different events ... ok 2021-02-11T10:15:56.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:56.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:56.731 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:56.769 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:56.769 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:56.982 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:57.025 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038556.717876_666.1","last_update":"2021-02-11T10:15:56.717876Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:57.025 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:15:57.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:57.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:57.026 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:57.066 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:57.066 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:57.342 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038556.717876_666.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:56.717876Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:57.342 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:15:57.342 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:15:57.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:57.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:57.343 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:57.386 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:57.387 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:57.639 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038556.717876_666.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:56.717876Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:57.640 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:15:57.640 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:57.640 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:57.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:57.640 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:57.681 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:57.682 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:57.932 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038556.717876_666.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:56.717876Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:57.933 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:57.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:57.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:57.933 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:57.973 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:57.973 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:58.221 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:58.264 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038556.717876_666.1","last_update":"2021-02-11T10:15:56.717876Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:58.265 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:58.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:58.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:58.265 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:58.306 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:58.306 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:58.563 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038556.717876_666.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:56.717876Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:58.563 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:58.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:58.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:58.823 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:58.861 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:58.861 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:59.082 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:15:59.124 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038558.817077_672.1","last_update":"2021-02-11T10:15:58.817077Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:59.125 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:59.125 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:59.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:59.125 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:59.165 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:59.165 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:59.414 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038558.817077_672.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:15:58.817077Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:59.414 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:16:03.894 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-88', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:03.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-88', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:03.895 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-88 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:16:03.945 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:03.946 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:04.148 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:04.146+0000 7f1c00d71080 1 bucket sync caught up with source: 2021-02-11T10:16:04.149 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000010.259.5, 00000000010.128.5, 00000000009.129.5, 00000000010.142.5, 00000000007.133.5, 00000000007.91.5, 00000000008.132.5, 00000000010.154.5, 00000000010.89.5, 00000000010.143.5, 00000000009.147.5] 2021-02-11T10:16:04.149 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000010.259.5, 00000000010.128.5, 00000000009.129.5, 00000000010.142.5, 00000000007.133.5, 00000000007.91.5, 00000000008.132.5, 00000000010.154.5, 00000000010.89.5, 00000000010.143.5, 00000000009.147.5] 2021-02-11T10:16:04.149 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:04.146+0000 7f1c00d71080 0 bucket checkpoint complete 2021-02-11T10:16:04.195 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "3" type: "OBJECT_CREATE" 2021-02-11T10:16:04.195 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "2" type: "OBJECT_CREATE" 2021-02-11T10:16:04.195 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "4" type: "OBJECT_CREATE" 2021-02-11T10:16:04.195 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "1" type: "OBJECT_CREATE" 2021-02-11T10:16:04.195 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "0" type: "OBJECT_CREATE" 2021-02-11T10:16:04.195 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "6" type: "OBJECT_CREATE" 2021-02-11T10:16:04.195 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "7" type: "OBJECT_CREATE" 2021-02-11T10:16:04.195 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "5" type: "OBJECT_CREATE" 2021-02-11T10:16:04.195 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "9" type: "OBJECT_CREATE" 2021-02-11T10:16:04.195 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2021-02-11T10:16:04.195 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "11" type: "OBJECT_CREATE" 2021-02-11T10:16:04.195 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "13" type: "OBJECT_CREATE" 2021-02-11T10:16:04.196 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "12" type: "OBJECT_CREATE" 2021-02-11T10:16:04.196 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "16" type: "OBJECT_CREATE" 2021-02-11T10:16:04.196 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "14" type: "OBJECT_CREATE" 2021-02-11T10:16:04.203 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "17" type: "OBJECT_CREATE" 2021-02-11T10:16:04.203 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "15" type: "OBJECT_CREATE" 2021-02-11T10:16:04.203 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "20" type: "OBJECT_CREATE" 2021-02-11T10:16:04.203 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "18" type: "OBJECT_CREATE" 2021-02-11T10:16:04.203 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "22" type: "OBJECT_CREATE" 2021-02-11T10:16:04.203 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "21" type: "OBJECT_CREATE" 2021-02-11T10:16:04.204 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "19" type: "OBJECT_CREATE" 2021-02-11T10:16:04.204 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "10" type: "OBJECT_CREATE" 2021-02-11T10:16:04.204 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "25" type: "OBJECT_CREATE" 2021-02-11T10:16:04.204 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "24" type: "OBJECT_CREATE" 2021-02-11T10:16:04.204 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "28" type: "OBJECT_CREATE" 2021-02-11T10:16:04.204 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "30" type: "OBJECT_CREATE" 2021-02-11T10:16:04.204 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "29" type: "OBJECT_CREATE" 2021-02-11T10:16:04.204 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "31" type: "OBJECT_CREATE" 2021-02-11T10:16:04.204 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "23" type: "OBJECT_CREATE" 2021-02-11T10:16:04.213 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "27" type: "OBJECT_CREATE" 2021-02-11T10:16:04.213 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "26" type: "OBJECT_CREATE" 2021-02-11T10:16:04.214 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "33" type: "OBJECT_CREATE" 2021-02-11T10:16:04.214 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "35" type: "OBJECT_CREATE" 2021-02-11T10:16:04.214 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "34" type: "OBJECT_CREATE" 2021-02-11T10:16:04.214 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "32" type: "OBJECT_CREATE" 2021-02-11T10:16:04.214 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "39" type: "OBJECT_CREATE" 2021-02-11T10:16:04.214 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "36" type: "OBJECT_CREATE" 2021-02-11T10:16:04.214 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "37" type: "OBJECT_CREATE" 2021-02-11T10:16:04.214 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "38" type: "OBJECT_CREATE" 2021-02-11T10:16:04.214 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "40" type: "OBJECT_CREATE" 2021-02-11T10:16:04.214 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "41" type: "OBJECT_CREATE" 2021-02-11T10:16:04.214 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "44" type: "OBJECT_CREATE" 2021-02-11T10:16:04.214 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "42" type: "OBJECT_CREATE" 2021-02-11T10:16:04.214 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "43" type: "OBJECT_CREATE" 2021-02-11T10:16:04.223 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "47" type: "OBJECT_CREATE" 2021-02-11T10:16:04.223 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "48" type: "OBJECT_CREATE" 2021-02-11T10:16:04.223 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "46" type: "OBJECT_CREATE" 2021-02-11T10:16:04.223 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "45" type: "OBJECT_CREATE" 2021-02-11T10:16:04.223 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "49" type: "OBJECT_CREATE" 2021-02-11T10:16:04.223 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "50" type: "OBJECT_CREATE" 2021-02-11T10:16:04.223 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "51" type: "OBJECT_CREATE" 2021-02-11T10:16:04.224 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "53" type: "OBJECT_CREATE" 2021-02-11T10:16:04.224 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "52" type: "OBJECT_CREATE" 2021-02-11T10:16:04.224 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "57" type: "OBJECT_CREATE" 2021-02-11T10:16:04.224 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "56" type: "OBJECT_CREATE" 2021-02-11T10:16:04.224 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "54" type: "OBJECT_CREATE" 2021-02-11T10:16:04.224 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "58" type: "OBJECT_CREATE" 2021-02-11T10:16:04.224 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "55" type: "OBJECT_CREATE" 2021-02-11T10:16:04.224 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "59" type: "OBJECT_CREATE" 2021-02-11T10:16:04.231 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "62" type: "OBJECT_CREATE" 2021-02-11T10:16:04.231 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "60" type: "OBJECT_CREATE" 2021-02-11T10:16:04.231 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "61" type: "OBJECT_CREATE" 2021-02-11T10:16:04.231 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "67" type: "OBJECT_CREATE" 2021-02-11T10:16:04.232 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "64" type: "OBJECT_CREATE" 2021-02-11T10:16:04.232 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "63" type: "OBJECT_CREATE" 2021-02-11T10:16:04.232 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "65" type: "OBJECT_CREATE" 2021-02-11T10:16:04.232 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "66" type: "OBJECT_CREATE" 2021-02-11T10:16:04.232 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "69" type: "OBJECT_CREATE" 2021-02-11T10:16:04.232 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "70" type: "OBJECT_CREATE" 2021-02-11T10:16:04.232 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "68" type: "OBJECT_CREATE" 2021-02-11T10:16:04.232 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "71" type: "OBJECT_CREATE" 2021-02-11T10:16:04.232 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "76" type: "OBJECT_CREATE" 2021-02-11T10:16:04.232 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "72" type: "OBJECT_CREATE" 2021-02-11T10:16:04.232 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "73" type: "OBJECT_CREATE" 2021-02-11T10:16:04.238 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "74" type: "OBJECT_CREATE" 2021-02-11T10:16:04.238 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "75" type: "OBJECT_CREATE" 2021-02-11T10:16:04.238 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "80" type: "OBJECT_CREATE" 2021-02-11T10:16:04.238 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "78" type: "OBJECT_CREATE" 2021-02-11T10:16:04.239 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "79" type: "OBJECT_CREATE" 2021-02-11T10:16:04.239 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "77" type: "OBJECT_CREATE" 2021-02-11T10:16:04.239 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "84" type: "OBJECT_CREATE" 2021-02-11T10:16:04.239 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "83" type: "OBJECT_CREATE" 2021-02-11T10:16:04.239 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "82" type: "OBJECT_CREATE" 2021-02-11T10:16:04.239 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "85" type: "OBJECT_CREATE" 2021-02-11T10:16:04.239 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "81" type: "OBJECT_CREATE" 2021-02-11T10:16:04.239 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "89" type: "OBJECT_CREATE" 2021-02-11T10:16:04.239 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "88" type: "OBJECT_CREATE" 2021-02-11T10:16:04.239 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "86" type: "OBJECT_CREATE" 2021-02-11T10:16:04.239 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "87" type: "OBJECT_CREATE" 2021-02-11T10:16:04.245 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "92" type: "OBJECT_CREATE" 2021-02-11T10:16:04.245 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "94" type: "OBJECT_CREATE" 2021-02-11T10:16:04.245 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "93" type: "OBJECT_CREATE" 2021-02-11T10:16:04.245 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "91" type: "OBJECT_CREATE" 2021-02-11T10:16:04.245 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "90" type: "OBJECT_CREATE" 2021-02-11T10:16:04.245 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "95" type: "OBJECT_CREATE" 2021-02-11T10:16:04.246 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "96" type: "OBJECT_CREATE" 2021-02-11T10:16:04.246 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "97" type: "OBJECT_CREATE" 2021-02-11T10:16:04.246 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "98" type: "OBJECT_CREATE" 2021-02-11T10:16:04.246 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "99" type: "OBJECT_CREATE" 2021-02-11T10:16:04.264 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T10:16:05.667 INFO:tasks.rgw_multisite_tests:test incremental fetching of events from a subscription ... ok 2021-02-11T10:16:05.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:05.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:05.667 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:05.712 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:05.712 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:05.932 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:16:05.962 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038565.646395_686.1","last_update":"2021-02-11T10:16:05.646395Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:16:05.963 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:16:05.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:05.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:05.963 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:06.004 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:06.005 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:06.262 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038565.646395_686.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:05.646395Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:06.262 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:16:06.262 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:16:06.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:06.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:06.263 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:06.307 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:06.307 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:06.573 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038565.646395_686.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:05.646395Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:06.573 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:16:06.573 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:16:06.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:06.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:06.574 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:06.614 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:06.614 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:06.857 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038565.646395_686.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:05.646395Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:06.857 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:16:06.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:06.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:06.858 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:06.897 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:06.897 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:07.115 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:16:07.158 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038565.646395_686.1","last_update":"2021-02-11T10:16:05.646395Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:16:07.158 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:16:07.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:07.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:07.159 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:07.199 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:07.199 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:07.448 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038565.646395_686.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:05.646395Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:07.448 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:16:08.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:08.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:08.309 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:08.344 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:08.344 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:08.556 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:16:08.605 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038568.303998_693.1","last_update":"2021-02-11T10:16:08.303998Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:16:08.605 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:16:08.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:08.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:08.606 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:08.639 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:08.640 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:08.884 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038568.303998_693.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:08.303998Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:08.885 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:16:09.323 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-89', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:09.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-89', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:09.323 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-89 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:16:09.380 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:09.380 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:09.588 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:09.586+0000 7f19a3a61080 1 waiting for incremental sync to catch up: 2021-02-11T10:16:09.588 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , 00000000001.139.5, 00000000001.282.5, 00000000001.151.5, , 00000000001.74.5, 00000000001.109.5, 00000000001.100.5, 00000000002.99.5] 2021-02-11T10:16:09.588 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000002.111.5, , 00000000001.139.5, 00000000001.282.5, 00000000001.151.5, , 00000000001.74.5, 00000000001.109.5, 00000000001.100.5, 00000000002.99.5] 2021-02-11T10:16:14.590 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:14.590+0000 7f19a3a61080 1 bucket sync caught up with source: 2021-02-11T10:16:14.590 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000002.111.5, , 00000000001.139.5, 00000000001.282.5, 00000000001.151.5, , 00000000001.74.5, 00000000001.109.5, 00000000001.100.5, 00000000002.99.5] 2021-02-11T10:16:14.590 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000002.111.5, , 00000000001.139.5, 00000000001.282.5, 00000000001.151.5, , 00000000001.74.5, 00000000001.109.5, 00000000001.100.5, 00000000002.99.5] 2021-02-11T10:16:14.590 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:14.590+0000 7f19a3a61080 0 bucket checkpoint complete 2021-02-11T10:16:14.629 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038569.118530.a2fd1310" 2021-02-11T10:16:14.629 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038569.150368.83fac85a" 2021-02-11T10:16:14.629 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038569.187912.1ae06257" 2021-02-11T10:16:14.629 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038569.302627.319f6f28" 2021-02-11T10:16:14.629 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038569.348679.b94ad24e" 2021-02-11T10:16:14.630 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038569.348940.d3a2d63d" 2021-02-11T10:16:14.630 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038569.427880.5c5637f6" 2021-02-11T10:16:14.630 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038569.558798.93859c58" 2021-02-11T10:16:14.630 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038569.698169.b2a2c873" 2021-02-11T10:16:14.630 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038569.698650.ff08994f" 2021-02-11T10:16:15.342 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T10:16:15.396 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1613038569.348940.d3a2d63d" 2021-02-11T10:16:15.396 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1613038569.427880.5c5637f6" 2021-02-11T10:16:15.396 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1613038569.558798.93859c58" 2021-02-11T10:16:15.396 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1613038569.698169.b2a2c873" 2021-02-11T10:16:15.396 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1613038569.698650.ff08994f" 2021-02-11T10:16:15.607 INFO:tasks.rgw_multisite_tests:test acking of some events in a subscription ... ok 2021-02-11T10:16:15.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:15.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:15.608 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:15.896 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:16:15.933 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038575.596347_696.1","last_update":"2021-02-11T10:16:15.596347Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:16:15.933 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:16:15.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:15.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:15.933 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:15.972 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:15.972 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:16.241 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038575.596347_696.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:15.596347Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:16.242 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:16:16.242 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:16:16.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:16.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:16.242 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:16.285 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:16.285 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:16.552 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038575.596347_696.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:15.596347Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:16.552 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:16:16.552 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:16:16.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:16.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:16.553 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:16.596 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:16.596 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:16.829 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038575.596347_696.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:15.596347Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:16.830 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:16:16.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:16.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:16.830 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:16.871 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:16.871 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:17.100 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:16:17.148 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038575.596347_696.1","last_update":"2021-02-11T10:16:15.596347Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:16:17.148 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:16:17.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:17.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:17.149 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:17.191 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:17.191 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:17.445 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038575.596347_696.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:15.596347Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:17.445 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:16:18.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:18.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:18.011 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:18.050 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:18.051 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:18.266 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:16:18.330 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038578.005188_702.1","last_update":"2021-02-11T10:16:18.005188Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:16:18.330 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:16:18.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:18.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:18.331 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:18.370 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:18.370 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:18.622 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038578.005188_702.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:18.005188Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:18.622 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:16:18.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-90', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:18.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-90', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:18.932 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-90 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:16:18.984 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:18.984 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:19.187 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:19.186+0000 7ff96efac080 1 bucket sync caught up with source: 2021-02-11T10:16:19.187 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [00000000001.167.5, , , , , , 00000000001.120.5, , , 00000000005.157.5, ] 2021-02-11T10:16:19.187 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [00000000001.167.5, , , , , , 00000000001.120.5, , , 00000000005.157.5, ] 2021-02-11T10:16:19.187 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:19.186+0000 7ff96efac080 0 bucket checkpoint complete 2021-02-11T10:16:19.223 DEBUG:tasks.rgw_multi.tests_ps:Event key: "multipart" type: "OBJECT_CREATE" 2021-02-11T10:16:19.224 DEBUG:tasks.rgw_multi.tests_ps:Event key: "put" type: "OBJECT_CREATE" 2021-02-11T10:16:19.224 DEBUG:tasks.rgw_multi.tests_ps:Event key: "copy" type: "OBJECT_CREATE" 2021-02-11T10:16:19.413 INFO:tasks.rgw_multisite_tests:test object creation notifications in using put/copy/post ... ok 2021-02-11T10:16:19.414 INFO:tasks.rgw_multisite_tests:test object creation s3 notifications in using put/copy/post on master ... ok 2021-02-11T10:16:19.414 INFO:tasks.rgw_multisite_tests:test multipart object upload on master ... ok 2021-02-11T10:16:19.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:19.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:19.415 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:19.452 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:19.452 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:19.650 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:16:19.691 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038579.402481_705.1","last_update":"2021-02-11T10:16:19.402481Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:16:19.691 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:16:19.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:19.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:19.691 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:19.731 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:19.731 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:20.005 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038579.402481_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:19.402481Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:20.005 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:16:20.005 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:16:20.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:20.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:20.006 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:20.045 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:20.046 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:20.305 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038579.402481_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:19.402481Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:20.305 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:16:20.306 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:16:20.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:20.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:20.306 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:20.347 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:20.347 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:20.590 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038579.402481_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:19.402481Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:20.591 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:16:20.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:20.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:20.591 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:20.632 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:20.632 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:20.862 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:16:20.903 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038579.402481_705.1","last_update":"2021-02-11T10:16:19.402481Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:16:20.903 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:16:20.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:20.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:20.904 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:20.943 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:20.943 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:21.193 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038579.402481_705.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:19.402481Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:21.194 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:16:21.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:21.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:21.795 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:21.829 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:21.829 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:22.033 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:16:22.081 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038581.789188_713.1","last_update":"2021-02-11T10:16:21.789188Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:16:22.082 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:16:22.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:22.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:22.082 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:22.115 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:22.115 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:22.357 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038581.789188_713.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:21.789188Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:22.357 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:16:22.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:22.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:22.493 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-91 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:16:22.540 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:22.541 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:22.766 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:22.765+0000 7f5e4a4a8080 1 bucket sync caught up with source: 2021-02-11T10:16:22.766 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , , , 00000000005.104.11] 2021-02-11T10:16:22.766 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , , , 00000000005.104.11] 2021-02-11T10:16:22.766 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:22.765+0000 7f5e4a4a8080 0 bucket checkpoint complete 2021-02-11T10:16:22.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:22.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:22.949 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-91 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:16:22.998 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:22.998 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:23.222 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:23.221+0000 7f4de5803080 1 bucket sync caught up with source: 2021-02-11T10:16:23.222 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , , , 00000000010.124.5] 2021-02-11T10:16:23.222 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , , , 00000000010.124.5] 2021-02-11T10:16:23.222 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:23.221+0000 7f4de5803080 0 bucket checkpoint complete 2021-02-11T10:16:23.267 DEBUG:tasks.rgw_multi.tests_ps:Event key: "foo" type: "OBJECT_DELETE" 2021-02-11T10:16:23.268 DEBUG:tasks.rgw_multi.tests_ps:Event key: "foo" type: "OBJECT_DELETE" 2021-02-11T10:16:23.268 DEBUG:tasks.rgw_multi.tests_ps:Event key: "foo" type: "OBJECT_DELETE" 2021-02-11T10:16:23.268 DEBUG:tasks.rgw_multi.tests_ps:Event key: "foo" type: "OBJECT_DELETE" 2021-02-11T10:16:23.274 DEBUG:tasks.rgw_multi.tests_ps:Event key: "foo" type: "DELETE_MARKER_CREATE" 2021-02-11T10:16:23.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:23.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:23.275 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-91 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:16:23.318 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:23.318 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:23.543 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:23.541+0000 7f1fe1012080 1 bucket sync caught up with source: 2021-02-11T10:16:23.543 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , , , 00000000010.124.5] 2021-02-11T10:16:23.543 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , , , 00000000010.124.5] 2021-02-11T10:16:23.543 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:23.541+0000 7f1fe1012080 0 bucket checkpoint complete 2021-02-11T10:16:23.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:23.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:23.582 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-91 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:16:23.631 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:23.631 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:23.869 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:16:23.865+0000 7fa9f671f080 1 bucket sync caught up with source: 2021-02-11T10:16:23.869 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , , , 00000000010.124.5] 2021-02-11T10:16:23.869 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , , , 00000000010.124.5] 2021-02-11T10:16:23.869 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:16:23.865+0000 7fa9f671f080 0 bucket checkpoint complete 2021-02-11T10:16:23.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:23.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:23.902 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-91 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:16:23.948 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:23.949 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:24.187 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:24.185+0000 7f54f0bba080 1 bucket sync caught up with source: 2021-02-11T10:16:24.187 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , , , 00000000010.124.5] 2021-02-11T10:16:24.188 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , , , 00000000010.124.5] 2021-02-11T10:16:24.188 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:24.185+0000 7f54f0bba080 0 bucket checkpoint complete 2021-02-11T10:16:24.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-91', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:24.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-91', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:24.223 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-91 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:16:24.271 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:24.272 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:24.516 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:16:24.513+0000 7f6da04ed080 1 bucket sync caught up with source: 2021-02-11T10:16:24.516 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , , , 00000000008.109.5] 2021-02-11T10:16:24.516 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , , , 00000000008.109.5] 2021-02-11T10:16:24.516 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:16:24.513+0000 7f6da04ed080 0 bucket checkpoint complete 2021-02-11T10:16:24.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-91', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:24.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-91', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:24.556 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-91 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:16:24.606 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:24.606 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:24.835 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:16:24.829+0000 7f120f326080 1 bucket sync caught up with source: 2021-02-11T10:16:24.835 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , , , 00000000008.109.5] 2021-02-11T10:16:24.835 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , , , 00000000008.109.5] 2021-02-11T10:16:24.835 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:16:24.829+0000 7f120f326080 0 bucket checkpoint complete 2021-02-11T10:16:24.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-91', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:24.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-91', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:24.871 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket erivkc-91 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:16:24.916 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:24.916 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:25.154 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:16:25.149+0000 7fe55b4be080 1 bucket sync caught up with source: 2021-02-11T10:16:25.155 INFO:teuthology.orchestra.run.smithi026.stderr: local status: [, , , , , , , , , , 00000000008.108.5] 2021-02-11T10:16:25.155 INFO:teuthology.orchestra.run.smithi026.stderr: remote markers: [, , , , , , , , , , 00000000008.108.5] 2021-02-11T10:16:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2021-02-11T10:16:25.149+0000 7fe55b4be080 0 bucket checkpoint complete 2021-02-11T10:16:25.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-91', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:25.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-91', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:25.188 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-91 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:16:25.234 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:25.235 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:25.463 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:25.461+0000 7f87c6c39080 1 bucket sync caught up with source: 2021-02-11T10:16:25.463 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, , , , , , , , , , 00000000008.108.5] 2021-02-11T10:16:25.463 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, , , , , , , , , , 00000000008.108.5] 2021-02-11T10:16:25.463 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:25.461+0000 7f87c6c39080 0 bucket checkpoint complete 2021-02-11T10:16:25.493 INFO:tasks.rgw_multi.zone_rados:comparing bucket=erivkc-91 zones={test-zone2, test-zone1} 2021-02-11T10:16:25.763 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:16:25.770 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:16:25.822 DEBUG:tasks.rgw_multi.zone_rados:o=foo 2021-02-11T10:16:25.882 CRITICAL:tasks.rgw_multi.zone_rados:key=foo is missing from zone=test-zone2 2021-02-11T10:16:25.883 DEBUG:tasks.rgw_multi.tests_ps:zonegroup_bucket_checkpoint failed, cannot delete bucket 2021-02-11T10:16:25.939 INFO:tasks.rgw_multisite_tests:test notification of deletion markers ... ok 2021-02-11T10:16:25.939 INFO:tasks.rgw_multisite_tests:test s3 notification of metadata on master ... ok 2021-02-11T10:16:25.939 INFO:tasks.rgw_multisite_tests:test s3 notification of tags on master ... ok 2021-02-11T10:16:25.940 INFO:tasks.rgw_multisite_tests:test s3 notification of object versions ... ok 2021-02-11T10:16:25.940 INFO:tasks.rgw_multisite_tests:test s3 notification of deletion markers on master ... ok 2021-02-11T10:16:25.940 INFO:tasks.rgw_multisite_tests:test pushing to http endpoint ... ok 2021-02-11T10:16:25.940 INFO:tasks.rgw_multisite_tests:test pushing to http endpoint s3 record format ... ok 2021-02-11T10:16:25.940 INFO:tasks.rgw_multisite_tests:test pushing to amqp endpoint ... ok 2021-02-11T10:16:25.941 INFO:tasks.rgw_multisite_tests:test pushing to amqp endpoint s3 record format ... ok 2021-02-11T10:16:25.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:25.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:25.941 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:25.983 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:25.983 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:26.199 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:16:26.242 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038581.789188_713.1","last_update":"2021-02-11T10:16:21.789188Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:16:26.242 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:16:26.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:26.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:26.243 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:26.282 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:26.283 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:26.522 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038581.789188_713.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:21.789188Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:26.522 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:16:26.522 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:16:26.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:26.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:26.523 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:26.569 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:26.570 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:26.794 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038581.789188_713.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:21.789188Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:26.794 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:16:26.794 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:16:26.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:26.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:26.795 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:26.836 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:26.837 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:27.067 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038581.789188_713.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:21.789188Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:27.067 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:16:27.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:27.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:27.068 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:27.109 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:27.110 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:27.324 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:16:27.364 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038581.789188_713.1","last_update":"2021-02-11T10:16:21.789188Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:16:27.364 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:16:27.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:27.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:27.364 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:27.405 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:27.406 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:27.652 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038581.789188_713.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:21.789188Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:27.652 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:16:28.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:28.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:28.463 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:28.499 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:28.499 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:28.710 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:16:28.750 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038588.457618_727.1","last_update":"2021-02-11T10:16:28.457618Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:16:28.750 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:16:28.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:28.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:28.751 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:28.793 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:28.793 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:29.042 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038588.457618_727.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:28.457618Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:29.043 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:16:29.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-92', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:29.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-92', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:29.507 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-92 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:16:29.559 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:29.559 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:29.797 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:29.797+0000 7f0388e2b080 1 bucket sync caught up with source: 2021-02-11T10:16:29.797 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000002.72.5, , 00000000001.125.5, 00000000001.165.5, 00000000001.169.5, , 00000000001.116.5, 00000000001.145.5, 00000000001.151.5, 00000000003.176.3] 2021-02-11T10:16:29.797 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000002.72.5, , 00000000001.125.5, 00000000001.165.5, 00000000001.169.5, , 00000000001.116.5, 00000000001.145.5, 00000000001.151.5, 00000000003.176.3] 2021-02-11T10:16:29.797 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:29.797+0000 7f0388e2b080 0 bucket checkpoint complete 2021-02-11T10:16:29.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-92', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:29.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'erivkc-92', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:16:29.954 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket erivkc-92 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:16:30.002 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:30.003 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:30.224 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:30.221+0000 7fcb3258d080 1 bucket sync caught up with source: 2021-02-11T10:16:30.224 INFO:teuthology.orchestra.run.smithi133.stderr: local status: [, 00000000004.76.5, , 00000000002.127.5, 00000000002.167.5, 00000000002.171.5, , 00000000002.118.5, 00000000002.147.5, 00000000002.153.5, 00000000005.180.5] 2021-02-11T10:16:30.224 INFO:teuthology.orchestra.run.smithi133.stderr: remote markers: [, 00000000004.76.5, , 00000000002.127.5, 00000000002.167.5, 00000000002.171.5, , 00000000002.118.5, 00000000002.147.5, 00000000002.153.5, 00000000005.180.5] 2021-02-11T10:16:30.224 INFO:teuthology.orchestra.run.smithi133.stderr:2021-02-11T10:16:30.221+0000 7fcb3258d080 0 bucket checkpoint complete 2021-02-11T10:16:30.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:30.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:30.346 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:30.386 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:30.386 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:30.598 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:16:30.645 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038590.335820_734.1","last_update":"2021-02-11T10:16:30.335820Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:16:30.645 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:16:30.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:30.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:30.646 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:30.680 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:30.680 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:30.941 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038590.335820_734.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:30.335820Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:30.941 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:16:30.949 WARNING:tasks.rgw_multi.tests_ps:superfluous records are found 2021-02-11T10:16:30.975 INFO:tasks.rgw_multisite_tests:test notification status upon bucket deletion ... ok 2021-02-11T10:16:30.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:30.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:30.976 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:31.016 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:31.016 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:31.276 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:16:31.315 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038590.335820_734.1","last_update":"2021-02-11T10:16:30.335820Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:16:31.315 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:16:31.315 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:31.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:31.315 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:31.355 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:31.355 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:31.818 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038590.335820_734.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:30.335820Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:31.818 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:16:31.818 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:16:31.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:31.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:31.819 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:32.117 INFO:teuthology.orchestra.run.smithi026.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038590.335820_734.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:30.335820Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:32.117 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:16:32.118 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:16:32.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:32.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:32.118 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:32.160 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:32.160 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:32.433 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038590.335820_734.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:30.335820Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:32.433 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:16:32.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:32.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:32.434 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:32.472 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:32.473 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:32.717 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:16:32.761 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038590.335820_734.1","last_update":"2021-02-11T10:16:30.335820Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:16:32.761 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:16:32.761 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:32.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:32.762 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:32.803 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:32.803 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:33.059 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038590.335820_734.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:30.335820Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:33.059 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:16:33.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:33.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:33.610 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:No --period given, using current period=f4d7bfad-1742-43a0-adcc-1b0cde4c6a67 2021-02-11T10:16:33.908 INFO:teuthology.orchestra.run.smithi026.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613038593.605434_741.1","last_update":"2021-02-11T10:16:33.605434Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:16:33.908 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:16:33.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:33.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:16:33.909 DEBUG:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:16:33.950 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:16:33.950 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:16:34.210 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f4d7bfad-1742-43a0-adcc-1b0cde4c6a67","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036985.070217_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:45.070217Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613038593.605434_741.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:16:33.605434Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:16:34.211 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:16:34.220 INFO:tasks.rgw_multi.tests_ps:missing topic is expected 2021-02-11T10:16:34.337 INFO:tasks.rgw_multisite_tests:test creating a subscription when no topic info exists ... ok 2021-02-11T10:16:34.337 INFO:tasks.rgw_multisite_tests:test updating topic associated with a notification ... ok 2021-02-11T10:16:34.338 INFO:tasks.rgw_multisite_tests:test updating the topic of a notification ... ok 2021-02-11T10:16:34.338 INFO:tasks.rgw_multisite_tests:test notification creation with multiple topics ... ok 2021-02-11T10:16:34.338 INFO:tasks.rgw_multisite_tests: 2021-02-11T10:16:34.338 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T10:16:34.338 INFO:tasks.rgw_multisite_tests:FAIL: test radosgw-admin commands 2021-02-11T10:16:34.338 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T10:16:34.338 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T10:16:34.339 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T10:16:34.339 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T10:16:34.339 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa/tasks/rgw_multi/tests_ps.py", line 3062, in test_ps_admin 2021-02-11T10:16:34.339 INFO:tasks.rgw_multisite_tests: assert_equal(len(parsed_result['events']), 10) 2021-02-11T10:16:34.339 INFO:tasks.rgw_multisite_tests:AssertionError: 11 != 10 2021-02-11T10:16:34.339 INFO:tasks.rgw_multisite_tests: 2021-02-11T10:16:34.339 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T10:16:34.339 INFO:tasks.rgw_multisite_tests:Ran 55 tests in 109.157s 2021-02-11T10:16:34.339 INFO:tasks.rgw_multisite_tests: 2021-02-11T10:16:34.339 INFO:tasks.rgw_multisite_tests:FAILED (failures=1) 2021-02-11T10:16:34.339 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa/tasks/rgw_multisite_tests.py", line 74, in begin raise RuntimeError(error_msg + 'test failures') RuntimeError: rgw multisite, rgw multisite pubsub, test failures 2021-02-11T10:16:34.377 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5e78bf5b7be64504b21fa81e52f676a4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa/tasks/rgw_multisite_tests.py", line 74, in begin raise RuntimeError(error_msg + 'test failures') RuntimeError: rgw multisite, rgw multisite pubsub, test failures 2021-02-11T10:16:34.377 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2021-02-11T10:16:34.435 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2021-02-11T10:16:34.445 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2021-02-11T10:16:34.459 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-02-11T10:16:34.459 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:16:34.462 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2021-02-11T10:16:34.457+0000 7f8a93fff700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 18762) UID: 0 2021-02-11T10:16:34.462 INFO:tasks.rgw.c1.client.0.smithi026.stdout:2021-02-11T10:16:34.457+0000 7f8ab81800c0 -1 shutting down 2021-02-11T10:16:34.651 INFO:tasks.rgw.c1.client.1.smithi026.stdout:2021-02-11T10:16:34.645+0000 7fd5a27fc700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5) 2021-02-11T10:16:34.717 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2021-02-11T10:16:34.717+0000 7f617cff9700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5) 2021-02-11T10:16:35.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-02-11T10:16:40.561 INFO:tasks.rgw.c1.client.0:Stopped 2021-02-11T10:16:40.562 DEBUG:teuthology.orchestra.run.smithi026:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2021-02-11T10:16:40.567 DEBUG:teuthology.orchestra.run.smithi026:> rm -f /home/ubuntu/cephtest/vault-token 2021-02-11T10:16:40.617 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-02-11T10:16:40.617 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:16:40.664 INFO:tasks.rgw.c1.client.1.smithi026.stdout:2021-02-11T10:16:40.661+0000 7fd676548700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 31419) UID: 0 2021-02-11T10:16:40.664 INFO:tasks.rgw.c1.client.1.smithi026.stdout:2021-02-11T10:16:40.661+0000 7fd68e0a30c0 -1 shutting down 2021-02-11T10:16:40.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-02-11T10:16:46.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-02-11T10:16:46.718 INFO:tasks.rgw.c1.client.1:Stopped 2021-02-11T10:16:46.718 DEBUG:teuthology.orchestra.run.smithi026:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2021-02-11T10:16:46.725 DEBUG:teuthology.orchestra.run.smithi026:> rm -f /home/ubuntu/cephtest/vault-token 2021-02-11T10:16:46.773 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-02-11T10:16:46.773 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:16:46.774 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2021-02-11T10:16:46.773+0000 7f62488b1700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 18718) UID: 0 2021-02-11T10:16:46.774 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2021-02-11T10:16:46.773+0000 7f626040c0c0 -1 shutting down 2021-02-11T10:16:51.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-02-11T10:16:52.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2021-02-11T10:16:52.875 INFO:tasks.rgw.c2.client.0:Stopped 2021-02-11T10:16:52.876 DEBUG:teuthology.orchestra.run.smithi133:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2021-02-11T10:16:52.883 DEBUG:teuthology.orchestra.run.smithi133:> rm -f /home/ubuntu/cephtest/vault-token 2021-02-11T10:16:52.930 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-02-11T10:16:52.930 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:16:52.973 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2021-02-11T10:16:52.973+0000 7f858092f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 19873) UID: 0 2021-02-11T10:16:52.973 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2021-02-11T10:16:52.973+0000 7f859848a0c0 -1 shutting down 2021-02-11T10:16:57.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2021-02-11T10:16:57.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2021-02-11T10:16:59.032 INFO:tasks.rgw.c2.client.1:Stopped 2021-02-11T10:16:59.033 DEBUG:teuthology.orchestra.run.smithi133:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2021-02-11T10:16:59.040 DEBUG:teuthology.orchestra.run.smithi133:> rm -f /home/ubuntu/cephtest/vault-token 2021-02-11T10:16:59.086 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2021-02-11T10:16:59.098 INFO:tasks.ceph.ceph_manager.c2:Canceling any pending splits or merges... 2021-02-11T10:16:59.099 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-02-11T10:16:59.442 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-02-11T10:16:59.443 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":68,"fsid":"4cba45bb-28f9-46d3-a27e-3c23d56a54ef","created":"2021-02-11T09:45:20.484106+0000","modified":"2021-02-11T10:02:05.425930+0000","last_up_change":"2021-02-11T09:45:28.077370+0000","last_in_change":"2021-02-11T09:45:28.077370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:45:28.093603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T09:45:29.100223+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2021-02-11T09:45:45.052625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2021-02-11T09:45:46.828886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2021-02-11T09:46:00.473080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2021-02-11T09:46:02.566743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2021-02-11T09:46:04.537722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2021-02-11T09:46:06.595255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"test-zone2.rgw.buckets.data","create_time":"2021-02-11T09:46:31.139307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"test-zone2.rgw.log","create_time":"2021-02-11T09:46:32.788384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":11,"pool_name":"test-zone2.rgw.control","create_time":"2021-02-11T09:46:34.578303+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":12,"pool_name":"test-zone2.rgw.meta","create_time":"2021-02-11T09:46:36.629574+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":13,"pool_name":"test-zone4.rgw.buckets.data","create_time":"2021-02-11T09:47:05.851929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":14,"pool_name":"test-zone4.rgw.log","create_time":"2021-02-11T09:47:07.764560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":15,"pool_name":"test-zone4.rgw.control","create_time":"2021-02-11T09:47:09.568705+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":16,"pool_name":"test-zone4.rgw.meta","create_time":"2021-02-11T09:47:11.643884+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":17,"pool_name":"test-zone2.rgw.buckets.index","create_time":"2021-02-11T09:49:24.854858+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":18,"pool_name":"test-zone4.rgw.buckets.index","create_time":"2021-02-11T09:49:24.890557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":19,"pool_name":"test-zone2.rgw.otp","create_time":"2021-02-11T09:54:21.661048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":20,"pool_name":"remove.rgw.log","create_time":"2021-02-11T10:01:58.320987+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":21,"pool_name":"remove.rgw.control","create_time":"2021-02-11T10:01:59.440855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":22,"pool_name":"remove.rgw.meta","create_time":"2021-02-11T10:02:01.487874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"1bd20326-5d00-4271-8124-7670b3b725ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":14635},{"type":"v1","addr":"172.21.15.133:6809","nonce":14635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":14635},{"type":"v1","addr":"172.21.15.133:6811","nonce":14635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":14635},{"type":"v1","addr":"172.21.15.133:6815","nonce":14635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":14635},{"type":"v1","addr":"172.21.15.133:6813","nonce":14635}]},"public_addr":"172.21.15.133:6809/14635","cluster_addr":"172.21.15.133:6811/14635","heartbeat_back_addr":"172.21.15.133:6815/14635","heartbeat_front_addr":"172.21.15.133:6813/14635","state":["exists","up"]},{"osd":1,"uuid":"2e90ab89-8f3b-459a-96ff-00d2d91bac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":14636},{"type":"v1","addr":"172.21.15.133:6817","nonce":14636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":14636},{"type":"v1","addr":"172.21.15.133:6819","nonce":14636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":14636},{"type":"v1","addr":"172.21.15.133:6823","nonce":14636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":14636},{"type":"v1","addr":"172.21.15.133:6821","nonce":14636}]},"public_addr":"172.21.15.133:6817/14636","cluster_addr":"172.21.15.133:6819/14636","heartbeat_back_addr":"172.21.15.133:6823/14636","heartbeat_front_addr":"172.21.15.133:6821/14636","state":["exists","up"]},{"osd":2,"uuid":"1b98a02b-858a-4100-8da2-2021ed431999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":14637},{"type":"v1","addr":"172.21.15.133:6801","nonce":14637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":14637},{"type":"v1","addr":"172.21.15.133:6803","nonce":14637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":14637},{"type":"v1","addr":"172.21.15.133:6807","nonce":14637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":14637},{"type":"v1","addr":"172.21.15.133:6805","nonce":14637}]},"public_addr":"172.21.15.133:6801/14637","cluster_addr":"172.21.15.133:6803/14637","heartbeat_back_addr":"172.21.15.133:6807/14637","heartbeat_front_addr":"172.21.15.133:6805/14637","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:45:26.337258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:45:26.374686+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:45:26.018929+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.13","mappings":[{"from":0,"to":2}]},{"pgid":"3.37","mappings":[{"from":0,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":0}]},{"pgid":"4.15","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":0}]},{"pgid":"4.26","mappings":[{"from":1,"to":0}]},{"pgid":"4.27","mappings":[{"from":1,"to":0}]},{"pgid":"4.3c","mappings":[{"from":1,"to":2}]},{"pgid":"10.12","mappings":[{"from":1,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T10:16:59.458 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2021-02-11T10:16:59.778 INFO:teuthology.misc:Shutting down mds daemons... 2021-02-11T10:16:59.779 INFO:teuthology.misc:Shutting down osd daemons... 2021-02-11T10:16:59.779 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2021-02-11T10:16:59.779 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:16:59.846 INFO:tasks.ceph.c2.osd.0:Stopped 2021-02-11T10:16:59.846 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2021-02-11T10:16:59.846 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:16:59.925 INFO:tasks.ceph.c2.osd.1:Stopped 2021-02-11T10:16:59.926 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2021-02-11T10:16:59.926 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:16:59.997 INFO:tasks.ceph.c2.osd.2:Stopped 2021-02-11T10:16:59.998 INFO:teuthology.misc:Shutting down mgr daemons... 2021-02-11T10:16:59.998 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2021-02-11T10:16:59.998 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:17:00.029 INFO:tasks.ceph.c2.mgr.x:Stopped 2021-02-11T10:17:00.029 INFO:teuthology.misc:Shutting down mon daemons... 2021-02-11T10:17:00.029 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2021-02-11T10:17:00.030 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:17:00.091 INFO:tasks.ceph.c2.mon.a:Stopped 2021-02-11T10:17:00.091 INFO:tasks.ceph:Checking cluster log for badness... 2021-02-11T10:17:00.092 DEBUG:teuthology.orchestra.run.smithi133:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2021-02-11T10:17:00.150 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi133.front.sepia.ceph.com 2021-02-11T10:17:00.151 DEBUG:teuthology.orchestra.run.smithi133:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2021-02-11T10:17:01.044 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi133.front.sepia.ceph.com 2021-02-11T10:17:01.044 DEBUG:teuthology.orchestra.run.smithi133:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2021-02-11T10:17:01.287 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi133.front.sepia.ceph.com 2021-02-11T10:17:01.287 DEBUG:teuthology.orchestra.run.smithi133:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2021-02-11T10:17:01.451 INFO:tasks.ceph:Archiving mon data... 2021-02-11T10:17:01.453 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874503/data/c2.mon.a.tgz 2021-02-11T10:17:01.453 DEBUG:teuthology.orchestra.run.smithi133:> mktemp 2021-02-11T10:17:01.459 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.bf77ea8z22 2021-02-11T10:17:01.459 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.bf77ea8z22 2021-02-11T10:17:01.872 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.bf77ea8z22 2021-02-11T10:17:01.932 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.bf77ea8z22 is 2MB 2021-02-11T10:17:02.011 DEBUG:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.bf77ea8z22 2021-02-11T10:17:02.018 INFO:tasks.ceph:Cleaning ceph cluster... 2021-02-11T10:17:02.018 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2021-02-11T10:17:02.020 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2021-02-11T10:17:02.074 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2021-02-11T10:17:02.084 INFO:tasks.ceph.ceph_manager.c1:Canceling any pending splits or merges... 2021-02-11T10:17:02.084 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-02-11T10:17:02.395 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-02-11T10:17:02.395 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":68,"fsid":"4594769f-0932-4e4a-a0a6-4220b143584b","created":"2021-02-11T09:44:27.643251+0000","modified":"2021-02-11T10:08:20.114423+0000","last_up_change":"2021-02-11T09:44:35.223423+0000","last_in_change":"2021-02-11T09:44:35.223423+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:44:35.233562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T09:44:48.853483+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2021-02-11T09:45:38.444153+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2021-02-11T09:45:41.006184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2021-02-11T09:45:50.607374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2021-02-11T09:45:51.740115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2021-02-11T09:45:53.700625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2021-02-11T09:45:55.753168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"test-zone1.rgw.buckets.data","create_time":"2021-02-11T09:46:13.566614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"test-zone1.rgw.log","create_time":"2021-02-11T09:46:14.952257+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":11,"pool_name":"test-zone1.rgw.control","create_time":"2021-02-11T09:46:16.718993+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":12,"pool_name":"test-zone1.rgw.meta","create_time":"2021-02-11T09:46:18.800445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":13,"pool_name":"test-zone1.rgw.otp","create_time":"2021-02-11T09:46:47.619124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":14,"pool_name":"test-zone3.rgw.buckets.data","create_time":"2021-02-11T09:46:48.406232+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":15,"pool_name":"test-zone3.rgw.log","create_time":"2021-02-11T09:46:49.998042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":16,"pool_name":"test-zone3.rgw.control","create_time":"2021-02-11T09:46:51.776571+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":17,"pool_name":"test-zone3.rgw.meta","create_time":"2021-02-11T09:46:53.855986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":18,"pool_name":"test-zone1.rgw.buckets.index","create_time":"2021-02-11T09:49:21.523054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":19,"pool_name":"test-zone3.rgw.buckets.index","create_time":"2021-02-11T09:49:24.885893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":20,"pool_name":"test-zone1.rgw.buckets.non-ec","create_time":"2021-02-11T10:08:17.986806+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"73968cb7-66da-4bd6-9b13-f436aa3f1654","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":14584},{"type":"v1","addr":"172.21.15.26:6801","nonce":14584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":14584},{"type":"v1","addr":"172.21.15.26:6803","nonce":14584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":14584},{"type":"v1","addr":"172.21.15.26:6807","nonce":14584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":14584},{"type":"v1","addr":"172.21.15.26:6805","nonce":14584}]},"public_addr":"172.21.15.26:6801/14584","cluster_addr":"172.21.15.26:6803/14584","heartbeat_back_addr":"172.21.15.26:6807/14584","heartbeat_front_addr":"172.21.15.26:6805/14584","state":["exists","up"]},{"osd":1,"uuid":"c3548235-9f89-49fd-a481-722c3fbad94c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":14582},{"type":"v1","addr":"172.21.15.26:6809","nonce":14582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":14582},{"type":"v1","addr":"172.21.15.26:6811","nonce":14582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":14582},{"type":"v1","addr":"172.21.15.26:6815","nonce":14582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":14582},{"type":"v1","addr":"172.21.15.26:6813","nonce":14582}]},"public_addr":"172.21.15.26:6809/14582","cluster_addr":"172.21.15.26:6811/14582","heartbeat_back_addr":"172.21.15.26:6815/14582","heartbeat_front_addr":"172.21.15.26:6813/14582","state":["exists","up"]},{"osd":2,"uuid":"8048fbe2-5786-4d39-83a4-853777450c19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":14583},{"type":"v1","addr":"172.21.15.26:6817","nonce":14583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":14583},{"type":"v1","addr":"172.21.15.26:6819","nonce":14583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":14583},{"type":"v1","addr":"172.21.15.26:6823","nonce":14583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":14583},{"type":"v1","addr":"172.21.15.26:6821","nonce":14583}]},"public_addr":"172.21.15.26:6817/14583","cluster_addr":"172.21.15.26:6819/14583","heartbeat_back_addr":"172.21.15.26:6823/14583","heartbeat_front_addr":"172.21.15.26:6821/14583","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:44:33.338318+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:44:33.190657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:44:33.469059+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.26","mappings":[{"from":0,"to":1}]},{"pgid":"3.34","mappings":[{"from":0,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]},{"pgid":"4.25","mappings":[{"from":1,"to":2}]},{"pgid":"4.30","mappings":[{"from":1,"to":2}]},{"pgid":"4.37","mappings":[{"from":1,"to":2}]},{"pgid":"10.1b","mappings":[{"from":2,"to":0}]},{"pgid":"14.13","mappings":[{"from":1,"to":0}]},{"pgid":"14.30","mappings":[{"from":1,"to":2}]},{"pgid":"14.3d","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T10:17:02.411 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2021-02-11T10:17:02.740 INFO:teuthology.misc:Shutting down mds daemons... 2021-02-11T10:17:02.740 INFO:teuthology.misc:Shutting down osd daemons... 2021-02-11T10:17:02.740 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2021-02-11T10:17:02.740 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:17:02.837 INFO:tasks.ceph.c1.osd.0:Stopped 2021-02-11T10:17:02.837 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2021-02-11T10:17:02.837 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:17:02.926 INFO:tasks.ceph.c1.osd.1:Stopped 2021-02-11T10:17:02.927 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2021-02-11T10:17:02.927 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:17:02.936 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2021-02-11T10:17:02.998 INFO:tasks.ceph.c1.osd.2:Stopped 2021-02-11T10:17:02.998 INFO:teuthology.misc:Shutting down mgr daemons... 2021-02-11T10:17:02.998 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2021-02-11T10:17:02.998 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:17:03.026 INFO:tasks.ceph.c1.mgr.x:Stopped 2021-02-11T10:17:03.026 INFO:teuthology.misc:Shutting down mon daemons... 2021-02-11T10:17:03.026 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2021-02-11T10:17:03.026 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:17:03.092 INFO:tasks.ceph.c1.mon.a:Stopped 2021-02-11T10:17:03.092 INFO:tasks.ceph:Checking cluster log for badness... 2021-02-11T10:17:03.093 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2021-02-11T10:17:03.157 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi026.front.sepia.ceph.com 2021-02-11T10:17:03.157 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2021-02-11T10:17:03.582 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi026.front.sepia.ceph.com 2021-02-11T10:17:03.583 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2021-02-11T10:17:03.746 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi026.front.sepia.ceph.com 2021-02-11T10:17:03.746 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2021-02-11T10:17:03.890 INFO:tasks.ceph:Archiving mon data... 2021-02-11T10:17:03.890 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874503/data/c1.mon.a.tgz 2021-02-11T10:17:03.891 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2021-02-11T10:17:03.896 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.cwUKxmiapA 2021-02-11T10:17:03.897 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.cwUKxmiapA 2021-02-11T10:17:04.311 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.cwUKxmiapA 2021-02-11T10:17:04.374 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.cwUKxmiapA is 2MB 2021-02-11T10:17:04.450 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.cwUKxmiapA 2021-02-11T10:17:04.457 INFO:tasks.ceph:Cleaning ceph cluster... 2021-02-11T10:17:04.457 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2021-02-11T10:17:04.500 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2021-02-11T10:17:04.515 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2021-02-11T10:17:04.516 DEBUG:teuthology.orchestra.run.smithi026:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2021-02-11T10:17:04.556 DEBUG:teuthology.orchestra.run.smithi133:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2021-02-11T10:17:04.573 INFO:teuthology.orchestra.run.smithi026.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2021-02-11T10:17:04.578 INFO:teuthology.orchestra.run.smithi133.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2021-02-11T10:17:04.586 INFO:tasks.ceph:Archiving crash dumps... 2021-02-11T10:17:04.588 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/crash to /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874503/remote/smithi026/crash 2021-02-11T10:17:04.589 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2021-02-11T10:17:04.641 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/lib/ceph/crash to /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874503/remote/smithi133/crash 2021-02-11T10:17:04.642 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2021-02-11T10:17:04.661 INFO:tasks.ceph:Compressing logs... 2021-02-11T10:17:04.661 DEBUG:teuthology.orchestra.run.smithi026:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-11T10:17:04.685 DEBUG:teuthology.orchestra.run.smithi133:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-11T10:18:33.963 INFO:tasks.ceph:Archiving logs... 2021-02-11T10:18:33.964 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/log/ceph to /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874503/remote/smithi026/log 2021-02-11T10:18:33.965 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/log/ceph -- . 2021-02-11T10:18:50.101 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/log/ceph to /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874503/remote/smithi133/log 2021-02-11T10:18:50.103 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar cz -f - -C /var/log/ceph -- . 2021-02-11T10:19:03.707 DEBUG:teuthology.run_tasks:Unwinding manager install 2021-02-11T10:19:03.720 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2021-02-11T10:19:03.720 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-02-11T10:19:03.722 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-02-11T10:19:03.771 DEBUG:teuthology.run_tasks:Unwinding manager clock 2021-02-11T10:19:03.781 INFO:teuthology.task.clock:Checking final clock skew... 2021-02-11T10:19:03.781 DEBUG:teuthology.orchestra.run.smithi026:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-11T10:19:03.802 INFO:teuthology.orchestra.run.smithi026.stdout: remote refid st t when poll reach delay offset jitter 2021-02-11T10:19:03.803 INFO:teuthology.orchestra.run.smithi026.stdout:============================================================================== 2021-02-11T10:19:03.803 INFO:teuthology.orchestra.run.smithi026.stdout:-hv01.front.sepi 216.240.36.24 3 u 4 64 377 0.074 -1.131 0.469 2021-02-11T10:19:03.803 INFO:teuthology.orchestra.run.smithi026.stdout:+hv02.front.sepi 163.237.218.18 2 u 55 64 377 0.098 0.570 0.333 2021-02-11T10:19:03.803 INFO:teuthology.orchestra.run.smithi026.stdout:+hv03.front.sepi 74.6.168.72 3 u 12 64 377 0.076 1.467 0.366 2021-02-11T10:19:03.803 INFO:teuthology.orchestra.run.smithi026.stdout:*hv04.front.sepi 144.172.126.201 3 u 63 64 377 0.070 0.839 0.370 2021-02-11T10:19:03.804 DEBUG:teuthology.orchestra.run.smithi133:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-11T10:19:03.823 INFO:teuthology.orchestra.run.smithi133.stdout: remote refid st t when poll reach delay offset jitter 2021-02-11T10:19:03.824 INFO:teuthology.orchestra.run.smithi133.stdout:============================================================================== 2021-02-11T10:19:03.824 INFO:teuthology.orchestra.run.smithi133.stdout:-hv01.front.sepi 216.240.36.24 3 u 60 64 377 0.094 -3.535 0.234 2021-02-11T10:19:03.824 INFO:teuthology.orchestra.run.smithi133.stdout:*hv02.front.sepi 163.237.218.18 2 u 53 64 377 0.083 -1.962 0.329 2021-02-11T10:19:03.824 INFO:teuthology.orchestra.run.smithi133.stdout:+hv03.front.sepi 74.6.168.72 3 u 3 64 377 0.078 -0.962 0.272 2021-02-11T10:19:03.824 INFO:teuthology.orchestra.run.smithi133.stdout:+hv04.front.sepi 144.172.126.201 3 u 59 64 377 0.067 -2.693 0.660 2021-02-11T10:19:03.825 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-02-11T10:19:03.836 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2021-02-11T10:19:03.837 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2021-02-11T10:19:03.855 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-02-11T10:19:03.871 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2021-02-11T10:19:03.888 INFO:teuthology.task.internal:Duration was 2444.416778 seconds 2021-02-11T10:19:03.889 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-02-11T10:19:03.906 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2021-02-11T10:19:03.906 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-02-11T10:19:03.908 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-02-11T10:19:03.958 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2021-02-11T10:19:03.959 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi026.front.sepia.ceph.com 2021-02-11T10:19:03.959 DEBUG:teuthology.orchestra.run.smithi026:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-02-11T10:19:03.970 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi133.front.sepia.ceph.com 2021-02-11T10:19:03.970 DEBUG:teuthology.orchestra.run.smithi133:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-02-11T10:19:03.983 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2021-02-11T10:19:03.983 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-11T10:19:04.017 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-11T10:19:04.056 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2021-02-11T10:19:04.066 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2021-02-11T10:19:04.067 DEBUG:teuthology.orchestra.run.smithi026:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-02-11T10:19:04.098 DEBUG:teuthology.orchestra.run.smithi133:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-02-11T10:19:04.114 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2021-02-11T10:19:04.126 DEBUG:teuthology.orchestra.run.smithi026:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-02-11T10:19:04.141 DEBUG:teuthology.orchestra.run.smithi133:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-02-11T10:19:04.152 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = core 2021-02-11T10:19:04.170 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = core 2021-02-11T10:19:04.188 DEBUG:teuthology.orchestra.run.smithi026:> test -e /home/ubuntu/cephtest/archive/coredump 2021-02-11T10:19:04.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T10:19:04.216 DEBUG:teuthology.orchestra.run.smithi133:> test -e /home/ubuntu/cephtest/archive/coredump 2021-02-11T10:19:04.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T10:19:04.234 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-02-11T10:19:04.244 INFO:teuthology.task.internal:Transferring archived files... 2021-02-11T10:19:04.245 DEBUG:teuthology.misc:Transferring archived files from smithi026:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874503/remote/smithi026 2021-02-11T10:19:04.245 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-02-11T10:19:04.282 DEBUG:teuthology.misc:Transferring archived files from smithi133:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874503/remote/smithi133 2021-02-11T10:19:04.283 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-02-11T10:19:04.303 INFO:teuthology.task.internal:Removing archive directory... 2021-02-11T10:19:04.304 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf -- /home/ubuntu/cephtest/archive 2021-02-11T10:19:04.325 DEBUG:teuthology.orchestra.run.smithi133:> rm -rf -- /home/ubuntu/cephtest/archive 2021-02-11T10:19:04.351 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2021-02-11T10:19:04.362 INFO:teuthology.task.internal:Not uploading archives. 2021-02-11T10:19:04.362 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2021-02-11T10:19:04.381 INFO:teuthology.task.internal:Tidying up after the test... 2021-02-11T10:19:04.382 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-11T10:19:04.384 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-11T10:19:04.389 INFO:teuthology.orchestra.run.smithi026.stdout: 524339 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 11 10:19 /home/ubuntu/cephtest 2021-02-11T10:19:04.400 INFO:teuthology.orchestra.run.smithi133.stdout: 524339 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 11 10:19 /home/ubuntu/cephtest 2021-02-11T10:19:04.402 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2021-02-11T10:19:04.427 INFO:teuthology.nuke:Checking targets against current locks 2021-02-11T10:19:04.505 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 412, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 269, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/nuke/__init__.py", line 266, in nuke for unnuked in p: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/nuke/__init__.py", line 280, in nuke_one keep_logs = ctx.keep_logs File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/config.py", line 255, in __getattr__ raise AttributeError(name) AttributeError: keep_logs